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Preface 


We  performed  this  research  for  the  Advanced  Systems  and  Technologies  (AS&T) 
Directorate  at  the  National  Reconnaissance  Office  (NRO).  Today,  the  NRO  faces  an 
operational  environment  that  is  faster  paced,  more  uncertain,  and  filled  with  more 
variables  than  it  was  even  ten  years  ago.  One  of  the  biggest  challenges  now  facing  the 
Intelligence  Community  (IC)  is  that  it  must  confront  unknown  threats  that  continue 
to  emerge  from  unexpected  directions. 

To  address  these  challenges,  the  NRO  asked  RAND  to  research  ways  that  AS&T 
and  the  NRO  could  be  become  more  responsive  toward  an  ever-changing  environ¬ 
ment.  To  do  this,  RAND  researched  three  different  topics,  each  designed  to  address 
a  different  component  within  the  NRO:  the  hardware,  the  people,  and  the  organiza¬ 
tion’s  processes.  For  the  hardware,  we  researched  the  benefits  of  modularity  and  devel¬ 
oped  a  list  of  factors  to  help  determine  whether  the  NRO’s  space  hardware  was  a  good 
candidate  for  a  modular  architecture.  For  the  people,  we  looked  at  how  other  occupa¬ 
tions  respond  to  unexpected  events  (i.e.,  surprise),  with  the  goal  of  identifying  a  set  of 
practices  that  could  be  employed  by  people  who  work  in  uncertain  environments.  To 
do  this,  we  spoke  with  ambassadors,  chief  executive  officers,  military  personnel,  and 
health  care  professionals,  and  we  report  on  some  common  methods  and  techniques 
that  they  use  to  prepare  for  and  respond  to  surprise.  Finally,  we  took  a  preliminary  look 
at  the  organizational  methods  used  inside  other  established  organizations  and  made 
some  observations  about  the  motives  behind  their  innovative  processes. 

The  findings  from  this  research  will  therefore  be  useful  for  NRO  strategists  as 
they  make  plans  to  shape  their  future  hardware  architectures,  workforce,  and  organi¬ 
zational  structures.  The  research  on  modularity  and  surprise  has  broader  applications 
beyond  the  NRO  and  the  IC  and  will  therefore  be  useful  for  individuals  working  on 
hardware  development  and  within  uncertain  environments,  respectively. 

This  research  was  conducted  within  the  Intelligence  Policy  Center  of  the  RAND 
National  Security  Research  Division  (NSRD).  NSRD  conducts  research  and  analy¬ 
sis  on  defense  and  national  security  topics  for  the  U.S.  and  allied  defense,  foreign 
policy,  homeland  security,  and  intelligence  communities  and  foundations  and  other 
nongovernmental  organizations  that  support  defense  and  national  security  analysis. 
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For  more  information  on  the  Intelligence  Policy  Center,  see  http://www.rand.org/ 
nsrd/ndri/centers/intel.html  or  contact  the  director  (contact  information  is  provided  on 
the  web  page). 

Comments  or  questions  on  this  report  should  be  addressed  to  the  project  leader, 
Dave  Baiocchi,  via  email  at  baiocchi@rand.org  or  phone  at  (310)  393-0411,  ext.  6658. 
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Summary 


The  U.S.  Intelligence  Community  (IC)  is  now  facing  a  larger  number  of  unknown 
threats  than  at  any  other  time  in  its  history.  During  the  Cold  War,  the  IC  faced  one 
primary,  well-identified  threat,  along  with  a  few  second-order  concerns.  Today,  the 
biggest  surprises  facing  the  IC  are  likely  to  come  from  places  of  which  the  community 
may  not  even  be  aware. 

To  help  address  these  challenges,  the  Advanced  Systems  and  Technology  (AS&T) 
Directorate  at  the  National  Reconnaissance  Office  (NRO)  asked  RAND  to  perform 
research  to  help  it  develop  strategic  plans  that  will  yield  insights  on  becoming  more 
flexible  and  adaptable.  We  settled  on  three  research  questions,  specifically  designed  to 
target  three  different  aspects  of  the  NRO  enterprise: 

•  How  can  the  NRO  build  more-flexible  hardware? 

•  How  can  NRO  personnel  become  better  prepared  to  deal  with  uncertainty? 

•  How  can  the  NRO’s  organizational  structures  be  used  to  promote  innovation  and 
creative  thought? 


How  Can  the  NRO  Build  More-Flexible  Hardware? 

To  investigate  this  question,  we  first  hypothesized  that  there  are  two  ways  to  build 
more-flexible  hardware:  (1)  by  building  in  excess  capability  and  (2)  by  using  a  modular 
architecture.  Excess  capability  gives  operators  the  freedom  to  develop  new  tactics,  tech¬ 
niques,  and  procedures  as  needs  change.  However,  it  can  be  challenging  to  convince 
decisionmakers  to  support  excess  capability  when  budgets  and  resources  are  becoming 
increasingly  constrained.  Therefore,  for  this  project,  we  investigated  the  suitability  of 
implementing  a  modular  architecture  for  the  NRO’s  space  systems. 

Modularity  Provides  Flexibility,  But  at  a  Cost 

Modularity  is  the  engineering  equivalent  of  a  financial  option:  Like  a  financial  option, 
modularity  permits  a  product  designer  to  invoke  some  flexibility  in  the  future  in 
exchange  for  a  cost  premium  that  is  paid  up  front.  For  space  systems,  this  premium  is 
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paid  in  the  form  of  additional  systems  engineering  that  is  needed  to  plan  and  design 
a  set  of  standardized  interfaces.  These  interfaces  must  be  designed  in  their  final  form 
at  the  onset  of  the  project  so  that  the  modules  are  ready  for  future  use,  providing  the 
potential  for  added  flexibility  and  responsiveness. 

However,  modular  systems  do  not  provide  all  this  flexibility  for  free.  Typically, 
a  modular  system  will  not  perform  each  function  as  well  as  the  equivalent  individual 
(singular)  systems.  For  example,  a  Swiss  Army  knife  allows  the  user  to  carry  a  number 
of  tools  around  in  one  small  package.  However,  this  flexibility  comes  at  a  price:  The 
tools  in  the  Swiss  Army  knife  will  never  perform  as  well  as  a  dedicated  knife,  cork¬ 
screw,  or  pair  of  scissors. 

Different  Classes  of  Systems  Provide  Different  Levels  of  Functionality  and  Benefit 

We  researched  several  examples  of  modular  systems  and  found  that  different  classes  of 
systems  provide  different  levels  of  functionality  and  benefits.  For  example,  dry-cell  AA 
batteries  and  carpet  are  designed  to  be  readily  scaled  based  on  user  needs,  but  the  pri¬ 
mary  functions  of  each  never  change.  In  contrast,  an  electronics  breadboard  with  resis¬ 
tors,  capacitors,  and  transistors  offers  nearly  infinite  functional  possibilities  to  the  user. 

However,  we  observed  that,  while  modular  systems  that  offer  changes  in  func¬ 
tion  are  certainly  more  flexible,  they  also  place  greater  responsibility  on  the  user.  For 
example,  in  order  to  use  a  breadboard  kit  to  build  an  electronic  device,  the  user  needs 
a  high  degree  of  knowledge  and  experience.  This  is  an  important  factor  that  designers 
should  weigh  when  considering  a  modular  architecture:  The  use  of  more-flexible  sys¬ 
tems  often  requires  more-knowledgeable  users. 

NRO  Space  Systems  Do  Not  Appear  to  Be  Strong  Candidates  for  Modularization 

Our  findings  suggest  that  some  systems  might  be  better  suited  for  modularity  than 
others.  To  apply  this  knowledge  about  modular  systems  to  the  NRO,  we  developed 
a  list  of  factors  to  help  system  designers  determine  if  a  system  is  a  good  candidate  for 
modularity. 

When  we  applied  our  factors  to  the  NRO’s  space-based  collection  systems,  we 
reached  an  inconclusive  result:  While  some  factors  seem  to  encourage  modularization, 
others  seem  to  discourage  it  or  are  neutral.  On  one  hand,  the  NRO  faces  uncertain 
future  user  needs,  along  with  a  customer  base  that  desires  a  highly  flexible  product. 
Both  of  these  factors  encourage  a  modular  architecture.  On  the  other  hand,  the  NRO 
relies  on  cutting-edge,  state-of-the-art  technologies  in  its  systems,  and  these  technolo¬ 
gies  do  not  lend  themselves  well  to  modularity.  This  is  because  rapid  changes  in  tech¬ 
nology  can  quickly  outgrow  the  static  interfaces  in  a  modular  architecture,  rendering 
the  entire  system  useless. 


Summary  ix 


The  NRO  Needs  to  Be  Able  to  Quantify  the  Value  of  Its  Intelligence-Gathering 
Systems 

So  what  can  be  done  to  move  forward  and  make  progress  toward  a  more  satisfying 
solution?  What  is  really  needed  to  provide  a  satisfying  answer  is  a  mathematical  rela¬ 
tionship  that  relates  desired  flexibility  with  the  likelihood  of  investment  gain  or  loss. 
To  gain  some  perspective,  we  looked  at  how  this  calculation  is  done  in  another  indus¬ 
try:  parking  garage  design.  The  parking  garage  designer  can  easily  quantify  the  balance 
between  flexibility  and  investment  risk.  Revenue  (in  dollars)  is  a  measure  of  value,  and 
an  interest  rate  is  used  to  determine  the  change  in  value  over  time. 

However,  there  is  an  important  difference  between  commercial  systems  and  the 
NRO’s  intelligence  systems:  It  is  very  difficult  to  evaluate  the  value  of  intelligence  sys¬ 
tems  and  how  that  value  changes  over  time.  This  observation  leads  to  a  key  conclusion: 
It  is  not  possible  to  find  the  optimum  “knee  in  the  curve”  for  implementing  modularity 
if  one  is  not  able  to  assess  the  value  of  the  intelligence  resulting  from  the  subject  system. 


How  Can  NRO  Personnel  Become  Better  Prepared  to  Deal  with 
Uncertainty? 

To  investigate  our  second  question,  we  started  by  thinking  about  other  professionals 
who  are  regularly  surrounded  by  uncertainty:  stock  traders;  U.S.  Navy  Sea,  Air,  Land 
(SEAL)  teams;  and  emergency  room  (ER)  doctors.  Practitioners  of  all  three  occupa¬ 
tions  must  be  comfortable  dealing  with  surprise,  and  this  idea  yielded  the  two  research 
questions  that  we  sought  to  address  in  this  work: 

•  Can  people  become  more  adept  at  planning  for  an  uncertain  future  by  studying 
surprise? 

•  Are  there  lessons  for  the  IC  in  how  different  professionals  respond  to  surprise? 

To  research  this  topic,  we  designed  a  framework  to  classify  different  professions 
based  on  the  following  two  factors:  (1)  how  quickly  they  typically  have  to  respond  to 
surprise  and  (2)  the  complexity  of  their  work  environment.  We  then  conducted  discus¬ 
sions  with  several  professionals  across  a  variety  of  fields  to  test  our  hypotheses. 

We  Identified  Two  Broad  Categories  of  Responses  to  Surprise  Among  Different 
Professions 

We  found  that  most  professionals  who  have  to  respond  to  surprises  within  seconds 
or  minutes  are  usually  skilled  in  touch  labor — i.e.,  they  work  with  their  hands.  This 
category  includes  surgeons,  Navy  SEALs,  test  pilots,  and  professional  athletes.  Prac¬ 
titioners  in  this  category  usually  must  control  feelings  of  fear  and  anxiety  when  they 
encounter  unexpected  events,  and  they  all  have  mental  and  physical  rituals  to  help 
them  manage  these  emotions. 
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Professionals  who  typically  have  more  time  to  respond  to  surprises  (e.g.,  hours, 
days,  or  weeks)  are  usually  valued  for  their  knowledge  capital.  This  category  includes 
chief  executive  officers  (CEOs),  ambassadors,  military  officers,  and  engineers.  When 
encountering  surprise,  these  practitioners  must  control  ego,  anger,  and  overreaction, 
and  the  most  successful  and  agile  practitioners  in  this  category  have  typically  devel¬ 
oped  mental  rituals  to  help  them  manage  these  specific  emotions. 

The  Level  of  Chaos  in  the  Environment  Also  Affects  People's  Response  to  Surprise 

We  found  that  the  level  of  chaos  in  the  environment  has  a  big  effect  on  how  people 
prepare  for  surprise.  For  example,  those  working  in  the  most  controlled  environments, 
such  as  an  athletic  stadium,  often  have  the  luxury  of  being  able  to  prepare  a  “what  if” 
plan  for  every  possible  unexpected  scenario  because  the  range  of  possibilities  is  discrete 
and  manageable.  We  found  that  professionals  working  in  moderately  chaotic  environ¬ 
ments  tend  to  develop  “what  if”  plans  for  the  most  likely  scenarios,  along  with  any  sce¬ 
nario  that  represents  an  existential  threat.  When  a  professional  of  this  sort  encounters 
something  in  the  environment  that  was  not  planned  for,  he  or  she  relies  on  experience 
or  training. 

The  Most  Complex  and  Chaotic  Situations  Are  Caused  by  Other  Humans,  Rather 
Than  Something  in  the  Environment 

Regarding  those  working  in  the  most  complex  environments,  we  arrived  at  an  unex¬ 
pected  observation:  All  the  individuals  working  in  the  most  complex  environments 
face  surprises  that  are  generated  by  other  humans.  A  CEO,  an  ambassador,  a  Special 
Weapons  and  Tactics  (SWAT)  team  captain,  a  Navy  SEAL,  and  military  general  offi¬ 
cers  all  fall  within  this  category.  We  found  that  all  of  these  professions  face  such  com¬ 
plex  operating  environments — with  an  infinite  number  of  things  that  can  go  wrong — 
that  it  does  not  make  sense  to  develop  comprehensive  “what  if”  plans.  Instead,  the 
successful  members  of  this  group  develop  generalized  frameworks  that  they  can  use  to 
deal  with  surprise,  regardless  of  the  specifics  of  the  surprise. 

The  Biggest  Surprises  Tend  to  Come  from  Third  Parties 

The  final  key  finding  from  our  research  on  surprise  is  that  the  biggest  surprises  are 
most  likely  to  come  from  third  parties — i.e.,  people  and  effects  outside  the  immediate 
field  of  view.  A  Navy  SEAL  was  the  first  to  make  this  point  to  us,  but  nearly  everyone 
else  made  the  same  observation. 

The  intuitive  reason  for  this  is  that  practitioners  often  spend  a  lot  of  time  think¬ 
ing  about  their  adversaries,  competitors,  or  key  challenges  and  therefore  develop  a  good 
understanding  of  how  these  forces  are  likely  to  behave.  One  way  to  address  the  threat 
of  the  unexpected  third  party  is  to  conduct  exercises  to  widen  the  organization’s  field 
of  view  and  highlight  potential  alternative  possibilities. 
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How  Can  the  NRO's  Organizational  Processes  Be  Used  to  Promote 
Responsiveness  and  Creative  Thought? 

Our  research  on  this  final  topic  was  motivated  by  the  following  objective:  How  did 
some  organizations  that  have  taken  steps  to  become  more  responsive  in  promoting 
innovation  and  creative  thought  achieve  this? 

We  looked  at  three  companies  suggested  to  us  by  NRO/AS&T:  Pfizer,  IBM,  and 
Caterpillar.  These  companies  have  all  been  recently  recognized  in  the  media  as  having 
gone  through  transformations  in  order  to  better  respond  to  pressures  in  the  market¬ 
place.  However,  each  company  reached  a  very  different  end  state:  Pfizer  become  more 
centralized,  IBM  started  selling  a  completely  different  product,  and  Caterpillar  became 
more  decentralized.  With  all  three  companies  looking  to  innovate,  why  did  they  take 
such  different  approaches? 

Innovation  Occurs  for  Many  Reasons,  Each  Requiring  a  Different  Approach 

We  found  that  innovation  occurs  for  many  reasons,  and  every  situation  requires  a  dif¬ 
ferent  approach.  For  example,  one  company  might  innovate  to  become  more  efficient 
(make  better  use  of  resources),  another  to  become  more  effective  (enhance  current 
capabilities),  and  a  third  to  become  more  agile  (quickly  adopt  new  technology).  The 
reason  for  the  innovation  will  help  determine  the  approach  taken. 

As  an  example,  we  found  that  Pfizer  decided  to  concentrate  on  anticancer  and 
Alzheimer’s  drugs.  To  do  this,  it  sold  off  and  divested  all  of  its  unrelated  properties  so 
that  it  could  concentrate  on  this  high-risk,  high-reward  goal.  In  the  process,  it  central¬ 
ized  its  organization  and  processes  to  pursue  a  single  mission. 

By  contrast,  Caterpillar  was  interested  in  becoming  more  responsive  to  its  custom¬ 
ers’  needs.  To  do  this,  the  company  decentralized  and  set  up  fully  contained  Caterpillar 
offices  around  the  country,  each  containing  everything  needed  to  run  the  business: 
product  experts,  sales  and  maintenance  teams,  and  finance  and  accounting  personnel. 
In  doing  this,  Caterpillar  was  able  to  customize  its  service  to  the  local  market,  but  this 
end  goal  required  a  different  approach  than  that  taken  by  Pfizer. 


Conclusions 

We  conclude  our  research  by  noting  that,  even  though  all  three  topics  appear  to  be  very 
different,  we  observed  three  common  lessons. 

Modularity  and  Innovation  Are  Not  Goals  in  Themselves 

The  first  observation  is  that  modularity  and  innovative  methods  are  not  goals  by 
themselves — they  are  tools  for  meeting  a  particular  goal.  Instead  of  saying  that  the 
organization  “needs  to  innovate”  or  “needs  to  implement  a  modular  architecture,” 
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strategists  should  first  set  the  priorities  and  the  mission  objectives.  Then  their  organiza¬ 
tion  will  be  in  a  position  to  determine  what  mechanisms  should  be  used  to  meet  the 
priorities. 

Strategic  Planning  Would  Be  Beneficial  for  All  Three  Areas  Discussed 

The  second  observation  is  that  success  in  modularity,  innovation,  and  reacting  to  sur¬ 
prise  all  benefit  from  at  least  a  partial  ability  to  predict  the  future.  Therefore,  we  con¬ 
clude  that  any  investments  in  developing  strategic  plans  or  visions,  along  with  exercises 
designed  to  probe  the  future,  can  advance  all  three  topics. 

Solutions  in  All  Three  Areas  Require  Not  Just  Hardware,  But  Also  People  and 
Organizational  Structures 

Modularity,  surprise,  and  innovative  processes  yield  ways  to  evolve  hardware,  people, 
and  corporate  structures,  respectively.  Merely  developing  flexible  hardware  will  not 
suffice  because  the  hardware  will  require  an  equally  flexible  staff  and  organizational 
structure  to  design,  implement,  and  operate  it. 
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Sea,  Air,  Land 

single-lens  reflex 

surface  warfare 

Special  Weapons  and  Tactics 
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CHAPTER  ONE 


Introduction 


NATIONAL  SECURITY  RESEARCH  DIVISION 


Increasing  Flexibility  &  Agility: 
Lessons  from  Modular  Design,  Occupational 
Surprise,  and  Commercial  Research  and 
Development  Processes 


The  National  Reconnaissance  Office  (NRO) — indeed,  the  entire  U.S.  Intelligence 
Community  (IC) — faces  an  operational  environment  that  is  faster  paced,  more  uncer¬ 
tain,  and  filled  with  more  variables  than  it  was  even  ten  years  ago.  One  of  the  biggest 
challenges  facing  the  IC  today  is  that  it  must  confront  unknown  threats  that  continue 
to  emerge  from  unexpected  directions.  This  represents  a  dramatic  contrast  to  the  envi¬ 
ronment  that  the  IC  faced  during  the  Cold  War,  where  the  challenges  were  (relatively 
speaking)  less  dynamic  and  better  understood.  As  one  way  of  addressing  this  problem, 
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the  NRO’s  Advanced  Systems  and  Technology  (AS&T)  Directorate  asked  RAND  to 
perform  research  on  ways  that  the  NRO  could  become  more  responsive. 

To  begin,  RAND  conducted  discussions  with  thought  leaders  throughout  the 
NRO,  and  together  RAND  and  the  NRO  developed  a  list  of  30  research  topics  for 
further  study.  AS&T  then  reviewed  this  list  and  picked  three  topics  that  RAND  was  to 
research  in  further  detail:  assessing  whether  a  modular  architecture  was  suitable  for  the 
NRO,  investigating  how  other  occupations  respond  to  unexpected  events  (and  what 
the  NRO  could  learn  from  these  professions),  and  performing  preliminary  research 
on  how  other  (nongovernment)  organizations  have  become  more  responsive  to  their 
customer  base. 

This  report  therefore  summarizes  the  work  that  RAND  performed  for  AS&T. 
The  report  is  divided  into  three  sections,  one  for  each  of  the  topics  that  are  mentioned 
above.  Each  section  serves  as  a  self-contained  module  in  which  we  outline  the  research 
objective,  method,  and  key  findings.  At  the  conclusion  of  this  report,  we  note  some 
observations  on  what  all  of  these  topics  have  in  common,  along  with  proposing  some 
new  ideas  for  continued  research. 
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This  Briefing  Summarizes  RAND’s  Research 
On  How  the  NRO  and  AS&T  Can  Become 

More  Agile 

Specific  research  questions: 


RAND 


Our  report  summarizes  RAND’s  research  on  how  the  NRO  and  AS&T  can  become 
more  agile  in  an  increasingly  complex  world.  To  address  this  overall  objective,  we 
researched  two  topics  in  detail: 

•  Would  the  NRO  benefit  from  building  modular  satellites? 

•  What  can  be  learned  about  becoming  more  responsive  by  looking  at  how  dif¬ 
ferent  occupations  (e.g.,  test  pilots,  surgeons,  chief  executive  officers  [CEOs]) 
respond  to  surprise? 

For  the  third  topic,  we  spent  time  doing  pilot  research  on  what  the  NRO  could 
learn  from  how  private-sector  companies  have  become  more  responsive.  We  will  discuss 
the  Endings  of  this  pilot  research  in  the  third  portion  of  this  report. 


CHAPTER  TWO 


Investigating  the  Suitability  of  Modularity  Toward  National 
Reconnaissance  Office  Space  Systems 


Our  Objective  Was  to  Determine  if  Modularity 
Was  Suitable  for  NRO  Space  Systems 

•  We  started  by  reviewing  the  academic  literature  and 
talking  to  subject  matter  experts  in  modularity: 

-  What  is  modularity? 

-  Why  do  it? 

-  What  are  the  ingredients  of  a  modular  architecture? 

•  We  developed  a  list  of  criteria  to  evaluate  whether 
systems  are  good  candidates  for  modularization 

-  We  tested  the  list  on  the  littoral  combat  ship  (LCS) 

•  How  do  NRO  space  systems  rate  using  these  criteria? 

•  What  questions  does  the  NRO  need  to  address  to  move 
forward? 

RAND 


We  will  begin  by  looking  at  the  first  topic:  modularity.  We  studied  modularity  because 
it  represents  one  approach  for  creating  systems  that  can  adapt  to  change.  This  slide 
describes  our  research  objective  and  method. 

As  we  will  show,  implementing  a  modular  architecture  provides  the  user  with  a 
system  that  is  more  easily  modified  to  respond  to  external  changes,  but  the  ability  to 
make  these  changes  often  comes  at  a  cost  of  additional  up-front  systems  engineering. 
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The  goal  for  our  analysis  was  to  take  a  first-order  look  at  whether  NRO  systems  are 
good  candidates  for  modularity. 

To  address  this  question,  we  started  by  examining  the  academic  literature  and 
consulting  with  subject  matter  experts  on  modularity.  We  then  used  our  findings  to 
form  a  clear  definition  of  what  modularity  is  and  how  it  can  make  a  system  more  easily 
adaptable  to  change.  After  researching  how  systems  can  adapt  to  change  using  modu¬ 
larity,  we  formed  an  outline  of  the  elements  that  are  necessary  for  a  successful  modular 
architecture. 

Next,  we  developed  a  list  of  first-order  factors  for  evaluating  whether  or  not  a 
system  is  a  good  candidate  for  modularization.  To  test  our  factors,  we  applied  them  to 
the  missions  addressed  by  the  littoral  combat  ship  (LCS),  which  is  widely  recognized 
as  being  a  modular  system  (Alkire  et  ah,  2007,  p.  6;  O’Rourke,  2012,  p.  1).  We  then 
applied  our  factors  to  NRO  space  systems  to  determine  if  they  might  benefit  from  a 
modular  architecture. 

This  section  concludes  with  recommended  actions  that  the  NRO  can  take  to  fur¬ 
ther  determine  how  to  incorporate  modularity  into  its  systems. 
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Modular  Systems  Are  Easily  Modified 
to  Address  Future  Needs 


Modularity  is  the  engineering  equivalent  of  a  financial  option: 
pay  a  premium  today  to  exercise  an  option  in  the  future. 


Modularity  allows  easier  ability 
to  dynamically  meet  customer 
demand 


Modularity  can  provide  flexibility 
functions  by  adding  components  to  a 
common  base,  but  that  system  will 
never  yield  the  best  performance 


RAND 

Knife  image  credit:  iStockphoto/Thinkstock 
Train  image  credit:  Hemera/Thinkstock 


Before  looking  explicitly  at  NRO  systems,  we  started  by  researching  the  basic  prin¬ 
ciples  of  modularity  to  understand  the  best  practices  of  operation.  This  slide  and  the 
following  two  slides  summarize  some  of  the  background  information  that  we  used  to 
inform  our  analysis.  We  present  this  information  to  provide  necessary  context  for  our 
later  analysis  of  NRO  space  systems. 

Modularity  combines  like  elements  or  components  into  independent  modules.  By 
doing  this,  the  modules  can  be  modified  separately,  and  in  this  way  a  modular  system 
can  quickly  adapt  to  changing  requirements  and  new  technology  (Baldwin  and  Clark, 
2000;  Lau  et  al.,  2011;  Sanchez  and  Mahoney,  1996;  Schilling,  2001;  and  Tassey, 
2000). 

One  useful  way  to  think  about  modularity  is  as  the  engineering  equivalent  of  a 
financial  option.  A  financial  option  is  a  monetary  instrument  that  gives  the  investor 
the  option  to  pay  an  up-front  cost  to  exercise  an  option  to  buy  or  sell  in  the  future.1  A 
financial  option  provides  value  through  its  ability  to  provide  choices  instead  of  forcing 
a  commitment.  Similarly,  in  engineering,  modularity  provides  an  option  (but  not  an 
obligation)  to  allow  for  change  in  the  future  (Baldwin  and  Clark,  2006). 


1  For  financial  options,  this  up-front  cost  may  be  in  the  form  of  increased  monetary  value  or,  alternatively, 
increased  risk. 
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In  practice,  modularity  can  manifest  itself  in  three  different  ways  (Baldwin  and 
Clark,  2000): 

•  Modularity  of  design  refers  to  modular  theory  that  is  used  during  the  design  and 
fabrication  phase,  in  which  components  are  developed  separately  and  in  parallel 
by  independent  designers.  Most  space  systems  are  designed  this  way;  for  example, 
the  sensing  package  (payload)  is  designed  separately  from  the  propulsion  or  navi¬ 
gation  systems.  Systems  that  have  modularity  of  design  may  or  may  not  end  up 
being  modular  for  the  end  user,  as  we  will  discuss  below. 

•  Modularity  of  production  refers  to  an  approach  in  which  all  of  the  functional 
components  are  manufactured  separately.  In  the  literature  that  we  reviewed,  the 
most  common  example  of  modularity  of  production  was  an  assembly  line,  where 
subcomponents  (often  sourced  from  different  suppliers)  are  brought  together  and 
assembled  into  a  larger  product.  Systems  that  have  modularity  of  design  often 
have  modularity  of  production  because  modular  designs  often  can  be  built  using 
modular  production  methods. 

•  Modularity  of  use  is  likely  what  most  people  think  of  when  considering  modular 
systems.  Systems  that  have  modularity  of  use  allow  the  end  user  to  exercise  the 
options  of  flexibility  that  we  discussed  earlier.  An  example  of  a  system  that  pro¬ 
vides  modularity  of  use  would  be  a  reconfigurable  messenger  bag  in  which  the 
user  can  remove  or  reposition  fabric  partitions  with  hook  and  loop  fasteners  to 
resize  internal  compartments. 

To  demonstrate  the  basic  benefits  and  challenges  of  modularity,  along  with  pro¬ 
viding  some  concrete  examples  of  the  differences  between  modularity  of  design,  con¬ 
sider  the  following  two  examples  of  modular  systems:  a  Swiss  Army  knife  and  a  train. 

The  Swiss  Army  knife  incorporates  a  large  number  of  tools  into  one  device,  and 
this  gives  the  user  an  increased  level  of  flexibility  for  responding  to  future  events.  All 
of  these  tools  mean  that  the  Swiss  Army  knife  has  many  purposes,  which  increases  its 
usefulness  to  the  end  user.  However,  while  a  Swiss  Army  knife  provides  a  variety  of  ser¬ 
viceable  instruments  and  tools  simultaneously,  none  of  these  tools  will  perform  as  well 
as  an  equivalent  singular  system,  such  as  a  dedicated  chef’s  knife,  corkscrew,  or  pair  of 
scissors.  The  Swiss  Army  knife  therefore  highlights  an  important  trade-off  that  is  often 
made  when  implementing  a  modular  architecture:  increased  flexibility  in  exchange  for 
nonoptimal  performance. 

The  Swiss  Army  knife  is  an  example  of  modularity  of  design  and  production,  but 
it  is  not  an  example  of  modularity  of  use.  All  of  the  tools  within  the  knife  operate  inde¬ 
pendently  of  one  another,  so  the  designers  (and  fabricators)  of  the  knife  design  each 
piece  separately,  optimizing  each  tool  for  the  volumetric  constraints  put  in  place  by  the 
knife’s  chassis.  This  approach  represents  modularity  in  design.  In  fabrication,  all  of  the 
tools  are  likely  made  on  separate  assembly  lines  and  are  combined  into  a  single  device 
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on  an  assembly  line.  This  is  modularity  of  production.  When  the  knife  reaches  the  end 
user,  the  customer  is  not  able  to  (easily)  add  new  tools  if  he  or  she  desires  new  func¬ 
tions  or  if  one  of  the  existing  tools  should  become  worn  or  fail.  In  short,  the  customer 
is  stuck  with  the  same  tools  that  came  with  the  knife.  Because  the  Swiss  Army  knife 
does  not  have  modularity  of  use,  it  is  therefore  a  good  example  of  an  object  that  can  be 
modular  through  the  design  and  fabrication  process  yet  not  modular  for  the  end  user. 

A  train  is  a  second  example  of  a  modular  system  because  the  cars  can  be  attached 
or  detached  easily  to  meet  dynamic  rider  demand.  This  system  allows  the  train  to  be 
easily  modified  to  respond  to  capacity  or  availability  needs  without  interfering  with  the 
overall  functionality  of  the  train.  In  addition,  should  one  car  need  repair,  it  can  easily 
be  removed  from  the  system  and  sent  to  the  repair  shop.  Trains  are  therefore  good 
examples  of  modularity  of  use  because  the  end  user  is  able  to  configure  the  train  cars 
in  a  way  that  helps  the  railway  operator  respond  to  changing  demand. 

The  train  example  is  also  useful  for  highlighting  a  secondary  point:  Many  modu¬ 
lar  systems  are  often  modules  of  modules.  The  train  car’s  wheel  module,  for  example, 
consists  of  the  wheels,  axes,  trucks,  and  brakes.  The  wheel  module  is  therefore  a  sub¬ 
component  of  the  overall  train  car,  which,  itself,  is  a  module. 

And  how  do  space-based  remote  sensing  systems  fit  in  with  the  types  of  modu¬ 
larity  that  we  outlined  above?  Space  systems  typically  have  modularity  of  design  and 
production  but  not  usually  modularity  of  use.  As  we  mentioned  above,  the  subsystems 
within  space  payloads  are  often  designed  and  fabricated  separately  by  independent 
engineering  teams.  The  advantage  to  this  approach  is  that  each  engineering  specialty 
(optical,  thermal,  mechanical,  propulsion,  etc.)  is  able  to  focus  its  expertise  to  design 
and  fabricate  modules  in  parallel  with  one  another.  Throughout  the  project,  a  system 
engineering  team  works  to  define  and  then  enforce  the  interfaces  between  the  mod¬ 
ules.  After  the  build  process  is  complete,  all  of  the  components  are  integrated  into  the 
final  package  and  tested  to  ensure  that  they  operate  as  a  single  unit. 

Now  that  we  have  highlighted  some  of  the  findings  from  our  literature  review  on 
modular  systems,  we  will  consider  how  the  NRO’s  space  systems  fit  into  this  context. 
We  will  draw  deeper  comparisons  in  the  coming  sections,  but  we  assumed  that  the 
NRO  is  most  interested  in  leveraging  modularity  of  use  to  provide  the  change  agents 
needed  to  respond  to  future  events.  We  make  this  assumption  with  the  understand¬ 
ing  that  the  NRO’s  current  space  systems  have  much  in  common  with  the  Swiss  Army 
knife:  They  rely  heavily  on  modularity  of  design  and  fabrication,  but  they  often  do  not 
provide  modularity  of  use  to  any  great  extent.  Therefore,  we  focused  our  subsequent 
research  (which  is  explained  on  the  next  slide)  to  look  at  methods  that  designers  can 
use  to  implement  or  improve  the  modularity  of  use. 
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Modularity  Offers  Different  Levels  of  Options  for  the 
End  User  to  Respond  to  New  Requirements 


Scale 

existing  functions 


Customize 
existing  features 

&l« 


Add 

new  functions 


Increasing  demand  for  user  knowledge  and  vision 
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When  considering  modular  effects  that  can  be  manipulated  by  the  end  user,  Ulrich 
and  Tung  (1991)  observed  that  there  are  varying  degrees  to  which  modularity  can  be 
leveraged  to  meet  specific  user  needs.  For  example,  some  products  only  allow  the  user 
to  proliferate  a  set  function,  while  other  products  give  the  user  complete  control  over 
the  functions  that  the  product  is  able  to  perform.  Using  Ulrich  and  Tung’s  observation 
as  a  starting  point,  we  developed  our  own  observations  on  how  modularity  allows  for 
different  types  of  changes.  Three  main  categories  are  described  below. 

The  first  example  of  flexibility  is  simply  scaling  the  existing  functionality,  in 
which  modularity  only  allows  for  changing  the  scope  or  size  without  allowing  the  user 
to  create  brand  new  functions.  Carpet  and  batteries  are  good  examples  of  this:  Both  are 
designed  to  be  readily  scaled  based  on  user  needs,  but  the  primary  functions  of  each 
never  change.  Batteries  just  provide  a  voltage,  but  that  voltage  is  a  variable  that  the  user 
can  vary  by  adding  (or  subtracting)  more  battery  modules  together.  Similarly,  carpet 
is  a  floor  covering,  and  the  user  can  change  the  square  footage  and  two-dimensional 
geometry  of  that  coverage.  In  both  cases,  though,  batteries  and  carpet  cannot  be  modi¬ 
fied  to  do  anything  beyond  their  primary  functions. 

The  next  example  is  using  modularity  to  customize  existing  functions  without 
dramatically  changing  the  primary  purpose  of  the  product.  For  example,  a  digital 
single-lens  reflex  (SLR)  camera  system  is  sold  alongside  an  array  of  lenses  representing 
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different  focal  lengths.  This  allows  the  user  to  exercise  greater  options  when  captur¬ 
ing  images,  but  the  interchangeable  lenses  do  not  change  the  primary  function  of  the 
camera,  which  is  to  take  photographs.  Another  example  is  bedding,  which  comes  in 
standard  sizes  (e.g.,  twin,  queen,  king).  This  standardization  allows  the  end  user  to 
easily  change  the  look  and  feel  of  the  bed  (perhaps  to  match  a  change  in  the  room’s 
paint  color)  using  fabrics  and  colors  from  different  manufacturers. 

The  most  robust  examples  in  the  slide  are  modular  systems  that  allow  the  user 
to  add  or  change  the  basic  functionality  of  the  device.  The  best  example  of  this  is  an 
electronics  breadboard  with  the  requisite  resistors,  capacitors,  chips,  and  wires.  This 
system  offers  the  user  an  infinite  number  of  options  to  change  and  adapt  to  future 
needs,  allowing  the  user  to  design  and  create  entirely  new  functionality  using  these 
components. 

One  key  observation  that  we  made  is  that  moving  from  left  to  right  on  the  chart 
in  the  slide  provides  more  flexibility  to  the  user,  but  it  also  places  a  greater  onus  on  the 
user  to  have  more  knowledge  and  vision  for  her  or  his  ultimate  needs.  Carpet  and  bat¬ 
teries  are  easy  to  use  and  install,  but  they  only  have  one  function.  By  contrast,  an  elec¬ 
tronics  breadboard  system  allows  for  nearly  infinite  functionality,  but  it  also  requires 
superior  knowledge  and  expertise  from  the  user.  In  addition,  the  most-flexible  systems 
require  greater  time  and  planning  to  design  and  implement.  This  observation  is  an 
important  one  to  consider  when  determining  the  level  of  flexibility  desired  in  a  modu¬ 
lar  architecture:  Greater  flexibility  places  more  demands  on  the  knowledge  base  of  the 
user  who  looks  to  exploit  the  resulting  architecture. 
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Modularity  Requires  Careful  Design  of 
Three  Key  Components 


Component 

Purpose 

Design  considerations 

Modules 

Provide  future  options, 
allows  separation  and 
reassembly 

•  What  current  and  future  functions  are  needed? 

•  What  are  the  performance  priorities? 

•  What  aspects  should  be  frozen/freed? 

•  How  to  protect  proprietary  information? 

Interfaces 

.....  .  ..  •  How  can  all  of  the  modules  be  accommodated? 

Allow  interaction  between  .  .  ,  ,  . 

,  ,  •  What  is  the  design  space  for  future 

modules  .  „ 

requirements? 

Test 

infrastructures 

Verifies  that  modules  will 
work  when  inserted  into  the 
system 

•  What  is  needed  to  ensure  a  module  will  work  as 
expected  when  inserted  into  the  system? 

•  Is  it  cost-effective  to  build  a  test  system? 

RAND 


Now  that  we  have  provided  background  information  on  what  it  means  to  design  and 
use  modular  systems,  it  is  worth  taking  a  more  detailed  look  at  the  three  key  compo¬ 
nents  of  any  modular  architecture:  the  modules,  interfaces,  and  test  infrastructure.  All 
of  these  components  are  recognized  in  the  literature  as  being  essential  to  modular  sys¬ 
tems,  but  they  are  described  nicely  in  Baldwin  and  Clark  (2000). 

Modules.  The  modules  are  the  elements  of  a  system  that  provide  the  freedom, 
allowing  users  to  swap  old  missions  or  obsolete  technologies  for  newer  ones.  Using 
physical  and/or  conceptual  separation  of  system  elements,  modules  can  be  altered 
without  disrupting  the  rest  of  the  system  components — a  key  feature  in  enabling  users 
to  adapt  to  new  changes  using  a  modular  architecture. 

When  designing  a  modular  architecture,  the  system  engineer  needs  to  consider 
some  key  questions  when  determining  how  to  partition  an  overall  system  into  indi¬ 
vidual  modules.  The  system  engineer  must  consider  what  functions  are  needed,  along 
with  how  those  functions  may  evolve  over  time.  The  answers  to  these  questions  will 
identify  what  aspects  of  the  system  should  be  partitioned  into  modules,  as  the  modules 
are  the  components  that  will  come  and  go  to  provide  those  change  agents  to  the  user. 
The  system  engineer  also  needs  to  consider  the  balance  between  reliability  and  perfor¬ 
mance:  Dividing  a  system  into  modules  will  add  interfaces  and  increase  the  number 
of  parts,  which  will  likely  increase  the  complexity  of  the  system.  In  some  cases,  com- 
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plicated  systems  have  more  parts  and  interdependencies,  which  may  make  them  less 
reliable. 

Interfaces.  The  interfaces  need  to  be  the  most  accommodating  part  of  the  system: 
As  modules  come  and  go,  the  interface  remains  embedded  within  the  system.  There¬ 
fore,  the  most  successful  interfaces  will  be  compatible  with  changing  technology,  even 
when  future  trajectories  of  the  technology  are  not  yet  known.  This  is  a  significant  chal¬ 
lenge  when  modularizing  a  system,  but  it  is  not  an  impossible  feat:  There  are  many 
examples  of  successful  interfaces  that  have  demonstrated  long  service  lifetimes. 

One  example  is  the  physical  interface  of  the  Universal  Serial  Bus  (USB)  plug, 
which  has  remained  consistent  for  over  20  years.  Despite  the  rapid  change  of  tech¬ 
nology,  USB  ports  still  use  the  same  physical  plugs,  are  backwards  compatible,  and 
demonstrate  the  possibility  of  anticipating  future  needs.  In  an  industry  known  for 
changing  every  18  months,  the  fact  that  USB  remains  the  primary  input/output  (I/O) 
interface  for  consumer  PCs  is  a  testament  to  the  original  architects’  ability  to  engineer 
a  very  flexible  interface. 

A  second  example  of  a  successful  interface  is  the  physical  plugs  associated  with 
electrical  alternating  current  (AC)  appliances.  While  each  country  does  have  a  differ¬ 
ent  standard,  each  of  these  plugs  has  been  very  robust:  Appliance  designers  have  relied 
on  the  same  physical  interface  with  the  U.S.  (consumer)  electrical  grid  since  the  early 
part  of  the  20th  century.  This  interface  is  now  just  as  adept  at  handling  a  modern  com¬ 
puter  as  it  was  with  a  1950s-era  toaster. 

Test  infrastructure.  A  test  infrastructure  is  the  third  component  of  a  successful 
modular  system.  The  test  infrastructure  is  a  way  to  ensure  that  new  modules  will  work 
as  expected  when  they  are  inserted  into  the  production  or  operational  system.  Develop¬ 
ing  a  test  infrastructure  usually  means  building  some  level  of  a  stand-alone  redundant 
system,  recognizing  that  this  will  incur  additional  cost.  The  alternative  to  this  is  to  plug 
modules  into  a  final  production  system,  but  this  defeats  the  purpose  of  having  a  modu¬ 
lar  architecture  because  the  production  system  will  serve  as  a  piece  of  test  equipment. 
If  the  new  module  does  not  work,  the  fielded  system  will  then  have  to  function  as  a 
testbed,  taking  the  system  away  from  its  regular  duties.  As  one  modularity  expert  with 
whom  we  spoke  noted,  if  the  method  for  testing  new  modules  is  just  plugging  them 
into  the  final  system,  there  is  no  advantage  to  building  a  modular  system  (Baldwin, 
2012). 

These  three  components  highlight  an  important  philosophical  approach  that  is 
inherent  with  modular  system  design:  Every  modular  architecture  must  “freeze”  some 
components  in  order  to  “free”  others  (Baldwin,  2012).  This  idea  is  evident  in  all  aspects 
of  modular  design:  Something  has  to  be  held  constant  in  order  to  let  other  components 
come  and  go.  This  is  a  powerful  idea  because  it  suggests  that  the  options  that  come 
with  modular  systems  do  not  come  without  cost.  In  freezing  components  (usually  the 
interfaces),  the  architect  is  putting  constraints  in  place,  but  these  constraints  are  (ironi¬ 
cally)  a  requirement  to  obtain  the  freedom  that  he  or  she  is  seeking. 
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All  of  the  background  presented  so  far  has  suggested  that  modular  architectures 
have  both  strengths  and  weaknesses.  This  implies  that  there  are  some  missions  or  needs 
that  may  be  more  suited  to  modularity  than  others,  and  we  set  out  to  develop  some  fac¬ 
tors  that  could  be  used  to  evaluate  whether  an  arbitrary  system  is  a  good  candidate  for 
modularity  of  use.  The  factors  that  we  propose  in  this  list  are  a  product  of  our  research 
and  thinking  on  modular  theory,  but  the  idea  of  developing  a  list  of  factors  was  moti¬ 
vated  by  Schilling  (2000)  and  Gershenson  et  al.  (2003). 

The  first  factor  is  associated  with  how  easy  it  is  to  physically  separate  the  functions 
that  will  later  be  assigned  to  different  modules.  We  use  the  word  component  to  describe 
these  functions,  and  in  using  this  word  we  refer  to  the  parts  that  make  up  the  final 
device  as  seen  from  the  end  user’s  perspective.2  If  the  system  can  be  easily  decomposed 
and  reassembled  via  some  components  that  roughly  map  to  different  functions,  the 
system  (or  the  mission  set)  is  a  good  candidate  for  modularity.  Conversely,  the  need  for 
highly  integrated  parts  makes  modularization  extremely  challenging.  This  is  because 
highly  integrated  parts  will  impose  a  high  cost  during  the  early  (and  subsequent)  sys¬ 
tems  engineering  processes,  as  engineers  deconflict  all  of  the  integrated  components 
to  determine  how  all  of  them  interact  with  one  another.  If  this  initial  cost  of  added 
systems  engineering  is  too  high,  it  may  overshadow  any  gains  in  flexibility  that  the 
modular  system  will  be  able  to  provide  further  down  the  road. 

The  second  factor  speaks  to  the  needs  of  the  user:  Does  the  user  prefer  flexibility 
or  highly  optimized  functionality?  As  we  mentioned  earlier,  modular  systems  have  the 
potential  to  provide  great  flexibility,  with  the  trade-off  being  that  modular  systems  will 
never  perform  as  well  as  singular  tools  designed  for  the  specific  tasks. 

The  third  factor  has  to  do  with  how  mature  the  technology  is.  Widely  used, 
mature,  or  universal  technologies  are  likely  to  have  existing  standards  in  place,  and 
this  makes  modularizing  these  technologies  less  challenging.  (This  practice  of  leverag¬ 
ing  universal  technologies  is  sometimes  referred  to  as  standards-based  innovation .)  By 
contrast,  emerging  technologies  will  have  less-mature  standards,  and  it  will  be  more 
challenging  to  design  a  set  of  enduring  interfaces.  For  this  reason,  emerging  and  spe¬ 
cialized  technology  is  more  difficult  to  modularize. 

The  fourth  factor  is  related  to  the  third:  If  the  technology  is  changing  in  a  pre¬ 
dictable,  incremental  way,  modularizing  will  allow  new  technology  to  be  easily  adapted 
(assuming  that  the  interfaces  have  been  designed  to  accommodate  these  changes.) 
However,  if  the  trajectory  is  unknown  or  revolutionary  changes  are  expected,  modu¬ 
larizing  could  actually  inhibit  innovation  by  tying  the  user  to  a  set  interface  that  will 
potentially  not  be  able  to  adapt  to  dramatic  changes  in  technology.  This  factor  moti¬ 
vates  an  important  observation:  Disruptive  products  are  unlikely  to  arise  out  of  modu- 


2  Of  course,  the  end  user  is  likely  to  have  very  little  knowledge  about  these  components,  but  that  is  unlikely  to 
affect  the  item’s  modularity  of  use.  For  example,  the  people  who  work  in  train  yards  coupling  and  uncoupling 
cars  only  need  to  understand  how  the  cars’  interfaces  work;  they  do  not  need  to  understand  the  details  of  the 
components,  such  as  the  wheel  assemblies,  passenger  compartments,  or  electrical  systems. 
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lar  systems  because  the  interfaces  (of  modular  systems)  will  likely  have  a  hard  time 
accommodating  such  dramatic  change. 

The  fifth  factor  assesses  the  level  of  uncertainty  in  future  user  needs.  If  the  future 
needs  are  well  defined,  this  factor  suggests  that  it  makes  more  sense  to  build  a  singu¬ 
lar  system  specifically  suited  for  the  particular  task.  By  contrast,  if  future  needs  are 
unknown,  modularity  is  a  suitable  way  to  provide  flexible  options. 

Taken  together,  applying  these  factors  to  a  system  or  mission  will  give  a  first-order 
indication  of  whether  a  modular  architecture  is  a  good  match. 
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The  LCS  is  a  Relevant  Example  Because  the 
U.S.  Navy  Shares  Similar  Goals  with  the  NRO 


Both  the  U.S.  Navy  and  NRO  are  looking  for  platforms  that  are 
flexible  and  capable  of  responding  to  a  variety  of  future  threat 

environments. 


Mission 

Primary  Function 

Antisubmarine  warfare 
(ASW) 

Detect,  track,  and  target 
enemy  submarines 

Mine  countermeasures  Hunt,  sweep,  and 

(MCM)  neutralize  mines 

Surface  warfare  (SUW) 

Counter  threats  from  small 
boats  via  human  capital- 
driven  means 

RAND 

LCS  images  courtesy  of  U.S.  Navy 


Before  we  show  how  NRO  space  systems  rate  using  our  factors,  we  will  show  the 
results  of  applying  our  factors  to  the  missions  serviced  by  the  U.S.  Navy’s  LCS.  This 
exercise  will  serve  as  a  “sanity  check”  to  demonstrate  that  our  factors  can  correctly 
identify  missions  that  would  be  well  served  by  a  modular  architecture. 

We  chose  the  LCS  for  three  reasons.  First,  the  LCS  is  recognized  as  having  been 
designed  to  be  modular  from  the  start  of  the  program;  indeed,  as  noted  in  a  2007 
RAND  report  on  the  ship,  “[m]odularity  is  at  the  heart  of  the  LCS  concept”  (Alkire  et 
ah,  2007,  p.  6).  Second,  the  LCS  is  a  system  that  is  being  procured  by  the  U.S.  Depart¬ 
ment  of  Defense  (DoD)  and  will  thus  serve  as  a  more  appropriate  comparison  to  the 
NRO’s  systems  than  evaluating  a  consumer  product,  such  as  a  messenger  bag  or  a  com¬ 
puter  peripheral.  (DoD  is  motivated  by  different  incentives  than  private  companies 
that  build  consumer  goods,  and  we  wanted  to  ensure  that  our  example  roughly  aligned 
with  the  IC’s  incentives  associated  with  protecting  national  security.)  Finally,  the  cre¬ 
ation  of  the  LCS  was  motivated  by  many  of  the  same  goals  that  the  NRO  currently 
has.  Specifically,  the  U.S.  Navy  built  the  LCS  because  it  was  looking  for  a  platform 
that  was  flexible  and  capable  of  responding  to  a  variety  of  future  threat  environments, 
and  we  heard  similar  language  when  asking  NRO/AS&T  personnel  about  their  future 
architecture  desires  (U.S.  Navy  personnel,  2012). 


Investigating  the  Suitability  of  Modularity  Toward  National  Reconnaissance  Office  Space  Systems  17 


Testing  These  Factors  on  LCS  Is  a  “Sanity  Check ” 
That  the  Factors  Can  identify  a  Modular  System 


What  factors  encourage 
modularization? 

Which  factors  discourage 
modularization? 

Components  are  easily  separable 

Components  are  highly  integrated 

Flexibility  valued  over  performance 

Highly-optimized  performance  desired 

Mature  technology 

Emerging  technology 

Incremental  changes  in  technology 

Dramatic  changes  in  technology 

Uncertainty  of  future  user  needs 

Well-defined  future  user  needs 

RAND 


By  testing  the  LCS  against  the  list  of  factors,  we  determined  that  the  missions 
serviced  by  the  LCS  are  good  candidates  for  a  modular  architecture.  Our  reasons  for 
making  these  assessments  are  described  in  more  detail  below,  and  we  base  our  judg¬ 
ments  on  data  that  we  collected  about  the  LCS  from  discussions  with  both  ship  opera¬ 
tors  in  San  Diego  and  personnel  in  the  LCS  program  element  office  in  Washington, 
D.C.  (U.S.  Navy  personnel,  2012). 

•  The  components  are  easily  separable.  At  the  start,  the  LCS  was  envisioned  to 
accommodate  technologies  that  had  been  previously  developed  for  other  DoD 
platforms.  The  designers  made  this  decision  because  they  knew  it  would  be  easier 
to  build  modules  based  on  existing  technology  rather  than  to  develop  state-of- 
the-art  modules  based  on  new  technologies.  This  is  one  of  the  key  trade-offs  that 
LCS  designers  made  when  building  the  system:  The  LCS  mission  packages  are 
indeed  modular,  but  the  components  used  in  the  mission  packages  are  based  on 
existing,  widely  available  technologies.  Because  the  designers  used  existing  tech¬ 
nology,  they  were  able  to  easily  separate  the  mission  packages  and  enclose  each 
mission  package  in  standard  volumetric  units  made  out  of  intermodal  containers 
(Conex  boxes). 
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•  Flexibility  in  functionality  is  desired.  As  we  mentioned  on  the  previous  slide, 
the  LCS  was  designed  to  flexibly  respond  to  a  variety  of  needs.  The  LCS  was 
designed  to  replace  legacy  (singular)  ships — specifically,  the  Avenger  class — that 
performed  the  mine  countermeasures  (MCM)  mission. 

•  The  LCS  designers  expect  incremental  changes  in  the  technologies  associ¬ 
ated  with  the  LCS  mission  packages.  As  an  example,  the  modular  hardware 
associated  with  the  surface  warfare  (SUW)  mission  includes  16  personnel,  two 
inflatable  boats,  and  a  few  heavy  machine  guns.  None  of  these  resources  will 
experience  such  significant  change  that  the  mission  package  will  no  longer  be 
accepted  by  the  LCS  seaframe. 

•  The  Navy  is  uncertain  to  what  extent  it  will  need  antisubmarine  warfare 
(ASW),  MCM,  and  SUW  capabilities  over  the  long  term.  Of  course,  this  factor 
depends  heavily  on  the  user’s  field  of  view  (FOV)  and  overall  context.  However, 
the  Navy  is  uncertain  what  percentage  of  antisubmarine,  antimine,  and  surface 
warfare  needs  it  will  be  called  upon  to  deliver  in  the  future.  The  Navy  person¬ 
nel  with  whom  we  spoke  noted  that  before  LCS,  the  Navy  used  different  ships 
for  the  MCM  and  SUW/ASW  missions.  In  addition,  they  noted  that  the  Navy 
has  not  been  called  upon  to  do  substantial  MCM  operations  very  frequently  over 
the  past  30  years.  This  uncertainty  means  that  the  LCS  offers  a  flexible  option 
because  it  allows  the  Navy  to  support  all  three  missions  using  one  piece  of  hard¬ 
ware  (assuming  that  the  Navy  has  time  to  swap  the  mission  packages  to  meet  the 
mission  need). 

We  were  unable  to  make  a  definitive  assessment  on  the  third  factor:  Is  the  tech¬ 
nology  on  LCS  mature  or  state  of  the  art?  In  our  conversations  with  LCS  personnel  in 
both  San  Diego  and  Washington,  D.C,  we  were  given  evidence  to  suggest  that  some 
of  the  components  within  the  LCS’s  mission  packages  were  based  on  existing  technol¬ 
ogy  used  elsewhere  in  DoD.  For  example,  the  SUW  package  has  two  30mm  guns  that 
were  originally  designed  for  use  on  a  Marine  Corps  personnel  transport  vehicle.  (The 
guns  that  have  been  repurposed  for  the  LCS  still  have  the  Marine  Corps  vehicle’s  seats 
attached,  despite  the  fact  that  these  seats  are  not  used  by  the  LCS  crew.)  In  another 
example,  LCS  personnel  noted  that  portions  of  the  computing  infrastructure  that  serve 
the  seaframe’s  software  are  commercially  available.  However,  despite  these  examples, 
we  recognize  that  the  LCS  does  have  extensive  custom-built  equipment,  especially  on 
the  technically  complex  ASW  module.  In  the  end,  we  did  not  have  enough  data  with 
which  to  make  a  decisive  determination  on  modularization  factor. 

Overall,  though,  four  of  the  five  factors  suggest  that  the  LCS  missions  are  well 
suited  for  a  modular  architecture,  and  we  conclude  that  our  factors  pass  our  “sanity 
check”  for  being  able  to  identify  missions  that  are  well  suited  for  modular  systems. 
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NRO  Systems  Have  Significant  Constraints  to 
Consider  When  Designing  a  New  Architecture 


NRO  system  constraints/ 
requirements 


Unique,  one-off  models 
State-of-the-art  technology 
Lack  of  physical  access 
Reliability  is  a  high  priority 
Secret/proprietary  design 

RAND 


Before  we  test  the  NRO  system  against  our  factors,  it  is  worth  noting  that  there  are 
some  additional  constraints  that  are  unique  to  NRO  space  systems. 

First,  each  NRO  mission  represents  a  unique,  “one-off”  design.  As  a  result,  few 
key  components  have  been  standardized.  If  the  system  is  going  to  be  modularized,  this 
lack  of  standardization  will  place  additional  burdens  on  the  system  engineer,  who  will 
have  to  develop  effective  interfaces  and  identify  all  of  the  interdependencies  between 
the  various  subsystems. 

NRO  systems  rely  on  state-of-the-art  sensing  technologies,  making  it  difficult 
to  determine  how  these  technologies  will  advance  in  the  future.  This  makes  grouping 
together  like-paced  elements  (for  the  purpose  of  containing  them  all  within  a  single 
module)  very  challenging. 

There  is  no  readily  available  physical  access  to  these  systems  once  they  are  deployed. 
This  not  only  inhibits  the  ability  to  test  on  the  actual  system,  but  this  lack  of  access  also 
makes  adequate  testing  prior  to  deploying  a  system  into  space  that  much  more  vital. 

NRO  space  systems  perform  a  critical  mission  to  the  nation  and,  therefore,  must 
remain  highly  reliable.  If  these  systems  are  going  to  be  modularized,  engineers  will 
have  to  devote  additional  resources  to  ensure  that  this  reliability  is  not  compromised. 

Finally,  NRO  systems  have  secret  or  proprietary  design  specifications,  and  modu¬ 
larizing  the  system  may  help  maintain  this  protection  of  information.  Protected  infor- 
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mation  can  be  encapsulated  in  a  module,  with  only  the  interface  communicating  with 
other  components  of  the  system. 
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Our  First-Order  Factors  Yield  Inconclusive 

Results  for  NRO  Space  Systems 

What  factors  encourage 
modularization? 

Which  factors  discourage 
modularization? 

Components  are  easily  separable 

Components  are  highly  integrated 

Flexibility  valued  over  performance 

Highly  optimized  performance  desired 

Mature  technology 

Emerging  technology 

Incremental  changes  in  technology 

Dramatic  changes  in  technology 

Uncertainty  of  future  user  needs 

Well-defined  future  user  needs 

To  first-order,  NRO  systems  do  not  lean  one  way  or  another  toward 
modularization.  So  how  do  we  move  forward  from  here? 

RAND 

This  slide  evaluates  the  first-order  suitability  of  NRO  space  systems  to  modularization. 

There  are  two  factors  that  seem  to  encourage  modularization:  The  NRO  desires 
highly  flexible  systems  that  are  able  to  adapt  to  meet  a  set  of  future  user  needs  that 
are  equally  uncertain.  Adopting  a  modular  architecture  would  allow  the  NRO  greater 
flexibility  to  change  user  functions  and  grant  the  NRO  the  agility  to  respond  more 
quickly  to  newer  technologies. 

There  are  also  two  factors  that  seem  to  discourage  modularization:  Current  NRO 
systems  are  highly  integrated  (because  of  the  space  constraints  imposed  by  the  launch 
vehicle),  and  these  systems  are  built  around  emerging  sensing  technologies.  The  prev¬ 
alence  of  emerging  technology  will  make  modularizing  these  systems  challenging 
because  of  a  lack  of  standardization  between  components. 

We  have  left  the  fourth  factor  as  neutral;  the  trajectory  of  the  technology  is  dif¬ 
ficult  to  predict,  and  the  NRO  systems  may  be  faced  with  either  technology  trajectory. 
In  addition,  we  had  a  hard  time  assessing  this  factor  ourselves  because  we  did  not  have 
enough  resources  (or  the  proper  access)  to  research  this  factor  in  detail.  Indeed,  even 
when  we  presented  this  factor  to  NRO/AS&T  personnel,  half  of  them  leaned  one  way 
and  the  other  half  leaned  the  other.  Even  if  we  had  been  able  to  make  an  assessment  on 
the  fourth  factor,  this  would  have  effectively  broken  the  tie  and  shifted  the  bias  one  way 
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or  another,  and  this  would  have  suggested  an  artificial  preference  toward  modularity  or 
singularity  that  does  not  really  exist. 

The  result  of  applying  these  factors  is  inconclusive:  To  first-order,  the  factors  sug¬ 
gest  that  the  NRO  space  systems  are  not  perfectly  suited  for  modularization  or  singu¬ 
larity.  How  could  we  break  the  tie  or  move  forward  to  determine  a  more  conclusive 
answer? 

The  first  way  to  move  past  the  impasse  would  be  identify  the  relative  weights 
assigned  to  each  of  these  factors.  We  recognize  that  some  of  the  factors  are  likely  to  be 
more  important  than  others,  and  the  first  step  for  AS&T  is  to  prioritize  which  factors 
are  most  important.  After  this  is  done,  the  result  might  provide  guidance  on  how  to 
break  the  tie,  especially  if  the  factors  that  encourage  modularization  are  determined 
to  be  weighted  more  heavily  than  the  others  (or  vice  versa).  If  flexibility  and  being  able 
to  respond  to  future  user  needs  are  most  important  to  the  NRO,  then  this  would  sug¬ 
gest  that  a  modular  architecture  is  best  suited  for  meeting  these  needs.  However,  it  is 
worth  recognizing  that  the  equivalent  modular  systems  are  likely  to  be  more  expensive 
and  complicated  than  the  existing  ones.  Likewise,  if  the  space  constraints  imposed  by 
the  launch  vehicle  and  the  desire  to  always  use  state-of-the-art  sensing  technologies  are 
most  important,  this  suggests  that  singular  systems  would  be  most  appropriate. 

One  consideration  that  we  have  not  mentioned  so  far  is  monetary  cost,  and  this 
suggests  another  way  to  break  the  tie.  So  far,  we  have  assumed  that  cost  is  what  deter¬ 
mines  the  initial  boundary  conditions  of  this  problem.  For  example,  launch  costs 
are  high,  and  this  is  what  encourages  the  current  practices  of  using  highly  integrated 
components. 

One  way  that  we  could  include  cost  in  these  factors  would  be  to  develop  a  math¬ 
ematical  device  that  relates  different  levels  of  flexibility  to  the  likelihood  of  losing  or 
gaining  money  across  the  lifetime  of  the  project.  To  investigate  this  idea,  we  turn  to 
an  example  industry  that  has  already  developed  such  a  device:  the  parking  garage 
industry. 
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There  Are  Three  Choices  When  Deciding  How  to 
Size  a  Parking  Garage  to  Meet  Demand 


A 


/  High  capacity  to  meet 
future  demand 
•S  Risk  of  unused  spaces 


B 


V  Lower  capacity  that 
meets  current  demand 
S  No  room  for  growth 


c 


v'  Low  initial  capacity 
meets  current  demand 
S  Spend  a  little  more  for 
the  option  to  add  on 
later 


RAND 


We  chose  the  parking  garage  example  because  it  offers  a  simple  analog  to  the  same 
problem  that  the  IC  is  trying  to  solve.  Specifically:  Is  there  a  way  to  determine  how 
much  more  should  be  spent  up  front  (if  anything)  buying  options  for  increased  flexibil¬ 
ity  in  the  future?  As  we  will  show,  parking  garage  designers  rely  on  a  simple  net  present 
value  (NPV)  calculation  to  quantify  the  balance  between  investment  risk  and  poten¬ 
tial  benefits  (de  Neufville  and  Scholtes,  2011).  After  we  describe  how  parking  garage 
designers  quantify  this  balance,  we  will  draw  parallels  to  the  NRO’s  space-based  col¬ 
lection  systems  and  identify  what  quantities  are  needed  to  repeat  this  approach  for 
these  systems.  However,  we  will  begin  by  describing  the  different  choices  that  a  park¬ 
ing  garage  developer  has  when  trying  to  size  a  new  garage  to  meet  expected  demand. 

Assuming  a  simple  scenario,  the  parking  garage  developer  has  three  choices  when 
determining  what  kind  of  garage  to  build.  The  developer  could  build  a  garage  that  is 
larger  than  what  she  or  he  needs  today  to  meet  current  demand  (option  A  in  the  slide). 
This  choice  represents  the  largest  up-front  investment,  but  it  also  provides  the  most 
excess  capacity  (and  therefore  additional  revenue)  should  the  demand  ever  increase. 
The  opposite  of  option  A  is  to  build  a  small  garage  that  exactly  meets  the  current 
demand  (option  B).  This  option  is  the  least  expensive,  but  it  provides  no  room  for 
growth.  If  the  demand  for  parking  spaces  increases,  the  developer  will  have  to  build  a 
new  garage.  The  final  option  is  to  build  a  garage  is  that  sized  for  today’s  demand  but 
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also  has  a  stronger  foundation  that  would  allow  the  developer  to  add  additional  floors 
should  the  demand  increase  in  the  future  (option  C). 
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The  Most-Flexible  Choice  Is  to  Pay 
For  the  Option  to  Build  More  Levels  Later 


B 


V  High  capacity  to  meet 
future  demand 

V  Risk  of  unused  spaces 


•s  Lower  capacity  that 
meets  current  demand 
s  No  room  for  growth 


c 


V  Low  initial  capacity 
meets  current  demand 
s  Spend  a  little  more  for 
the  option  to  add  on 
later 
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The  most-flexible  choice  is  to  pay  for  the  option  to  build  more  levels  later.  Option  C  will 
be  less  expensive  than  A  and  more  expensive  than  B,  but  it  allows  the  garage  developer 
to  spend  more  today  for  the  option  to  add  extra  floors  later  to  meet  future  demand.  So 
how  does  the  garage  developer  decide  exactly  what  dimensions  he  or  she  needs,  based 
on  his  or  her  appetite  associated  with  the  risk  of  investment  loss  (along  with  the  desire 
for  potential  gain)?  Or,  to  pose  a  question  that  is  often  asked  when  designing  modular 
systems:  Where  is  the  “knee”  in  the  modularity  curve?  How  many  options  should  the 
developer  purchase  to  achieve  a  suitable  level  of  future  flexibility? 
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The  Parking  Garage  Designer  Can  Quantify  the 
Balance  Between  Flexibility  and  Investment  Risk 

•  What  can  we  learn  from  the  method  used  to  design 
parking  garages? 

expected  demand 
construction,  mx  costs 
revenue 
interest  rate 

•  Result:  relates  likelihood  of  gaining  or  losing  $N  for 
a  desired  level  of  flexibility. 

•  When  compared  to  a  rigid  design,  a  flexible  garage 
offers  the  ability  to  better  match  future  demand  with 
less  likelihood  of  taking  a  loss. 
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So  how  does  the  parking  garage  developer  decide  how  to  proceed?  The  answer  is  that 
the  developer  performs  an  NPV  calculation  that  allows  him  or  her  to  quantify  the  bal¬ 
ance  between  future  flexibility  and  likelihood  of  investment  gain  or  loss  (de  Neufville 
and  Scholtes,  2011). 

There  are  four  key  quantities  that  the  developer  uses  to  perform  this  calculation: 

The  first  quantity  is  the  expected  future  demand;  this  is  represented  by  a  prob¬ 
ability  distribution. 

The  developer  also  considers  the  construction  and  maintenance  costs  across  the 
life  of  the  garage. 

The  developer  determines  how  much  to  charge  each  car  that  enters  the  garage, 
and  this  is  used  to  calculate  revenue,  given  a  particular  demand  curve. 

Finally,  the  developer  uses  an  interest  rate  to  normalize  price  across  the  lifetime  of 
the  project  and  determine  the  NPV. 

If  the  developer  chooses  a  single-input  demand  probability,  he  or  she  can  calculate 
an  NPV  across  the  lifetime  of  the  project  for  options  A,  B,  and  C.  To  make  this  cal¬ 
culation  more  useful,  the  developer  could  perform  a  Monte  Carlo  calculation,  varying 
the  demand  curve  each  time.  The  result  of  this  calculation  will  provide  the  developer 


->  Net  Present  Value 
(NPV) 
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with  a  set  of  curves  that  relate  a  given  level  of  flexibility  with  the  likelihood  of  gaining 
or  losing  $N  across  the  lifetime  of  the  project. 

This  is  exactly  the  type  of  relationship  that  we  are  seeking  to  develop  for  NRO 
space  systems,  so  what  would  it  take  to  repeat  this  calculation  for  the  NRO? 
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The  Parking  Garage  Highlights  the  Difficulty 
in  Quantifying  Flexibility  of  Intelligence  Systems 


Metric 

Parking  Garage 

Satellite 

Comments  on  Satellite  Case 

Demand 

Probability  distribution 
(#  spots/yr) 

Probability  distribution 

Costs 

O&M,  upgrades 
(dollars) 

Knowable 

(dollars) 

Value 

Revenue 
(Price  per  spot) 

? 

How  does  the  IC  assess  “value”?  By  number 
of  collects?  What  metric  can  assess  the 
value  of  intelligence? 

Effect  that  time 
has  on  value 

Interest  rate 

? 

How  much  more  is  a  collect  worth  today 
than  a  week  from  now?  Or  a  year  from  now? 

Total  lifetime  value 

Likelihood  of  profit 
(Dollars) 

Likelihood  of  gain/loss 

Observation:  An  early  step  in  becoming  more  flexible  is  determining 
how  to  measure  the  value  of  the  intelligence  collection  systems. 
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This  table  relates  the  quantities  used  for  the  parking  garage  calculation  to  what  would 
be  needed  to  repeat  this  calculation  for  intelligence-gathering  satellites.  The  first  two 
metrics,  demand  and  costs,  might  be  challenging  to  obtain  for  a  satellite,  but  an  expe¬ 
rienced  practitioner  should  be  able  to  make  a  reasonable  estimate  on  what  the  demand 
and  costs  would  be  for  building  such  a  system. 

A  larger  problem  arises  when  asked  to  consider  the  value  of  an  intelligence¬ 
gathering  system.  The  value  of  the  parking  garage  is  assessed  using  revenue  (number 
of  dollars  collected  per  parked  car);  what  is  the  equivalent  of  this  for  the  IC?  How  does 
the  IC  assess  value?  In  addition,  what  metric  is  used  to  determine  the  effect  that  time 
has  on  value?  What  was  the  value  of  an  overhead  image  of  the  bin  Laden  compound  a 
day  before  the  raid?  What  would  the  value  have  been  for  the  same  image  a  week  later? 
The  parking  garage  example  therefore  highlights  the  difficulty  in  quantifying  the  flex¬ 
ibility  of  intelligence  systems. 

We  recognize  that  we  are  not  making  a  new  observation  when  we  note  that  devel¬ 
oping  metrics  to  assess  the  value  of  intelligence  systems  is  very  difficult.  However,  we 
propose  that  this  is  exactly  what  is  required  to  solve  the  problem  of  balancing  cost  with 
flexibility  to  assess  the  suitability  of  modular  systems.  Without  being  able  to  assess  the 
value  of  an  intelligence  system,  it  is  not  possible  to  develop  a  calculation  that  tells  the 


Investigating  the  Suitability  of  Modularity  Toward  National  Reconnaissance  Office  Space  Systems  29 


user  when  it  makes  sense  to  modularize  or  not,  or  to  what  degree  a  system  should  be 
modularized. 
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We  Recommend  Two  Approaches  to 
Make  Further  Progress  on  This  Topic 

•  Our  criteria  suggest  inconclusive  results  on  whether  to 
pursue  modularization. 

•  Moving  forward,  there  are  two  options: 

-  To  properly  determine  how  much  flexibility 
(modularity)  is  needed,  a  system  architect  needs  to 
be  able  to  quantify  the  value  of  intelligence. 

-  In  the  meantime,  a  more  qualitative  (although  less 
satisfying)  approach  would  be  to  identify  2  to  4 
performance  needs  that  really  matter: 

•  Is  it  possible  to  build  modules  that  support  the 
performance  objectives? 

•  What  interfaces  will  be  needed? 

RAND 


We  conclude  this  section  by  noting  our  key  finding:  NRO  space  systems  are  not  obvi¬ 
ous  candidates  for  modularization.  So  what  can  be  done  to  proceed? 

The  parking  garage  analogy  yielded  one  suggestion  for  making  progress:  Develop 
methods  to  assess  the  value  of  space-based  intelligence-gathering  systems.  This  is  the 
best  way  to  progress  toward  developing  a  quantitative  approach  to  answering  the  ques¬ 
tion  that  most  system  architects  ask  when  considering  a  modular  design:  Where  is  the 
knee  in  the  modularity  curve?  Or,  how  much  modularity  provides  the  most  flexibility 
for  a  modest  up-front  investment?  As  our  analogy  to  the  parking  garage  demonstrated, 
getting  a  quantitative  answer  to  these  questions  means  solving  the  problem  of  quanti¬ 
fying  the  value  of  the  intelligence  provided  by  the  system. 

We  recognize  that  quantifying  the  value  of  intelligence  sensors  is  one  of  the  IC’s 
biggest  challenges.  NRO/AS&T  is  currently  working  on  this  problem  for  its  own  sen¬ 
sors,  but  it  recognizes  that  more  work  is  needed  before  an  effective  solution  is  developed. 

In  the  meantime,  we  recommend  a  secondary  approach  for  moving  forward.  As 
a  starting  point,  the  modularity  experts  with  whom  we  spoke  recommended  thinking 
about  what  could  be  done  if  starting  from  scratch,  without  any  restrictions  (Baldwin, 
2012;  Schilling,  2012).  What  would  the  NRO  design  to  do  today’s  mission,  if  it 
were  allowed  to  ignore  legacy  missions  or  capabilities  that  are  being  carried  forward 
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as  forced  requirements?  What  key  performance  metrics  are  captured  by  this  “from 
scratch”  system? 

After  identifying  the  appropriate  mission  needs,  the  NRO  could  then  consider 
the  benefits  of  modularity  by  determining  if  these  needs  would  be  easily  modularized 
using  the  factors  that  we  presented  earlier.  Does  it  make  sense  to  build  modules  to  sup¬ 
port  these  needs?  Is  it  even  possible,  given  the  existing  technology  trajectories?  What 
interfaces  will  be  needed? 

We  recognize  that  the  results  from  this  approach  will  never  be  as  satisfying  as 
the  results  from  solving  the  IC’s  analog  to  the  parking  garage  problem  because  it  will 
not  yield  a  quantitative  result.  Instead,  it  offers  an  interim  solution  for  making  prog¬ 
ress  before  the  larger  challenge  of  addressing  intelligence  value  is  addressed.  However, 
even  though  this  approach  offers  an  easier  path  (in  return  for  less-satisfying  answers), 
there  are  still  several  challenges  that  must  be  addressed.  Specifically,  identifying  the 
right  components  to  modularize  to  accommodate  change  will  be  a  significant  chal¬ 
lenge,  and  modularizing  when  there  are  many  unknowns  can  be  precarious.  There  is 
a  significant  amount  of  risk  in  committing  to  any  specific  trajectory  or  performance 
metric  when  the  direction  of  the  technology  is  uncertain,  and  for  this  reason  modular¬ 
izing  too  early  in  the  development  of  a  technology  is  often  not  recommended.  If  the 
technology  trajectory  is  misidentihed,  the  user  could  be  left  with  an  obsolete  system, 
having  wasted  effort  on  modularizing  and  subsequently  finding  it  to  be  incompatible 
with  newly  evolved  technology. 

An  example  from  the  music  industry,  which  was  brought  to  our  attention  by  one 
of  the  modularity  experts  with  whom  we  spoke,  is  helpful  for  illustrating  this  point 
(Schilling,  2012).  Up  until  the  mid-1990s,  the  technologies  used  for  music  playback 
all  progressed  with  increases  in  playback  fidelity,  starting  with  vinyl  records  before 
moving  to  cassette  tapes  and  then  compact  discs  (CDs).  In  the  ’90s,  the  music  industry 
started  looking  for  a  playback  medium  that  was  of  even  higher  fidelity  than  the  com¬ 
pact  disc,  and  they  started  working  on  a  standard  called  Super  CD.  Yet,  Super  CDs  do 
not  exist  today.  What  happened? 

The  reason  that  Super  CDs  have  not  been  commercialized  is  because  the  music 
industry  got  it  wrong:  They  thought  that  consumers  wanted  higher-fidelity  sound,  but 
consumers  really  wanted  portability.  While  the  industry  was  trying  to  settle  on  the 
standards  for  the  Super  CD,  the  rise  of  inexpensive  personal  computing  gave  rise  to 
MP3,  an  electronic  hie  format.  What  is  interesting  about  this  example  is  that  the  pro¬ 
gression  from  records  to  cassettes  to  CDs  aligns  with  increasing  improvements  in  both 
fidelity  and  portability,  but  the  music  industry  either  failed  to  identify  this  parallelism 
or  simply  followed  the  wrong  metric.  The  lesson  is  that  predicting  future  needs  is  not 
straightforward  and  is  highly  uncertain.  Choosing  the  wrong  parameters  and  technol¬ 
ogy  trajectories  can  end  up  inhibiting  innovation  and  enhancement  of  a  system  and,  in 
the  worst  case,  can  leave  the  user  with  a  system  with  functions  she  or  he  does  not  need 
and  lacking  the  functions  that  are  needed. 
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Occupational  Surprise 


This  Briefing  Summarizes  RAND’s  Research 
on  How  the  NRO  and  AS&T  Can  Become 

More  Agile 

Specific  research  questions: 


RAND 


We  now  move  on  to  the  second  research  topic:  What  can  we  learn  about  agility  by 
looking  at  how  different  occupations  respond  to  surprise? 
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Can  the  Way  These  People  Respond  to  Surprise 
Provide  Insights  on  How  to  Become  More  Agile? 
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Surprise  is  universal.  Virtually  all  professionals  must  plan  for  and  deal  with  unexpected 
events  as  part  of  their  daily  practice.  This  makes  “occupational”  surprise  an  attractive 
study  topic  because  the  lessons  promise  to  apply  broadly  across  a  variety  of  professions, 
including  the  U.S.  IC. 

Our  work  began  with  two  basic  questions  about  how  people  react  to  surprise: 


1.  How  can  we  become  more  adept  at  planning  for  an  uncertain  future? 

2.  Are  there  lessons  in  how  different  occupations  respond  to  surprise? 


Our  research  objective  for  this  task  was  therefore  to  investigate  how  different 
occupations  respond  to  unexpected  events. 

We  started  our  research  by  considering  a  list  of  occupations,  some  examples  of 
which  are  shown  in  this  slide.  We  started  our  list  with  professions  like  National  Football 
League  (NFL)  coaches,  Navy  SEALs,  and  test  pilots,  but  we  also  realized  that  longer- 
term  thinkers  like  CEOs  and  ambassadors  also  have  to  deal  with  the  unexpected. 

After  identifying  a  starting  list  of  professions,  we  consulted  the  academic  literature 
to  identify  a  basic  decisionmaking  model.  We  then  built  on  that  model,  developing  a 
framework  that  would  allow  us  to  more  easily  characterize  and  classify  the  professions. 
We  used  the  framework  to  help  us  develop  our  hypotheses  as  we  started  our  research. 
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Before  we  describe  our  method  and  findings  in  detail,  it  is  worth  noting  that  this 
research  topic  proved  to  be  so  rich  that  we  have  also  published  a  stand-alone  docu¬ 
ment  that  more  completely  describes  this  research  (Baiocchi  and  Fox,  forthcoming). 
For  the  purpose  of  this  report,  we  will  outline  our  initial  hypotheses,  research  method, 
and  findings.  We  will  conclude  by  making  observations  on  how  our  findings  can  be 
used  by  NRO/AS&T.  Our  longer  document  greatly  expands  on  these  topics,  includ¬ 
ing  additional  context  on  our  hypotheses  and  data  collection  methods,  along  with 
additional  anecdotes  that  we  gathered  from  all  of  our  practitioners.  In  addition,  we 
expanded  the  context  of  our  findings  and  recommendations  so  they  could  be  accessible 
to  an  audience  outside  the  U.S.  IC. 

Finally,  at  the  request  of  our  sponsor,  we  purposely  did  not  include  occupations 
from  the  IC  in  our  assessment.  We  did  this  for  two  reasons.  First,  we  did  not  want 
to  bias  our  result  by  including  our  immediate  audience  in  our  research.  If  we  had, 
we  may  have  favored  conclusions  toward  one  group  or  another,  possibly  favoring  one 
conclusion  over  another.  Second,  the  sponsor  was  interested  in  presenting  the  results 
from  this  research  to  the  IC  at  large,  and  we  suspect  that  different  parts  of  the  IC  will 
fall  into  different  portions  of  our  framework.  Leaving  the  IC  out  of  our  initial  dataset 
means  that  each  group  can  make  its  own  assessment  on  where  it  stands  within  our 
framework  and  conclusions. 

At  the  most  basic  level,  preparing  for  and  responding  to  surprises  is  about  effec¬ 
tive  decisionmaking.  Therefore,  our  initial  approach  was  to  begin  by  reviewing  the 
pertinent  decision-science  literature  that  relates  to  unexpected  events. 

There  are  a  number  of  books  that  address  aspects  of  the  surprise  problem.  Weick 
and  Sutcliffe’s  Managing  the  Unexpected:  Resilient  Performance  in  an  Age  of  Uncertainty 
(2007)  presents  a  set  of  rules  that  high-reliability  organizations  (HROs)  should  follow 
in  order  to  effectively  mitigate  the  negative  effects  associated  with  unexpected  events.1 
Bazerman  and  Watkins’s  Predictable  Surprises:  The  Disasters  You  Should  Have  Seen 
Coming,  and  How  to  Prevent  Them  (2008),  focuses  on  the  nature  of  the  surprise  itself, 
and  they  outline  a  set  of  characteristics  that  are  useful  for  identifying  “predictable  sur¬ 
prises”  before  they  cascade  into  more-damaging  effects.  In  Thinking  in  Time:  The  Uses 
of  History  for  Decision-Makers  (1986),  Neustadt  and  May  focus  on  the  utility  of  using 
historical  analogies  to  inform  present-day  decisionmaking.  McCall,  Lombardo,  and 
Morrison’s  Lessons  of  Experience:  How  Successful  Executives  Develop  on  the  Job  (1998) 
discusses  how  important  experience  is  for  effectively  responding  to  unexpected  events. 

As  we  would  later  confirm,  all  of  the  components  highlighted  in  these  works — 
strategies  for  developing  effective  response  techniques,  methods  for  identifying  surprise 
indicators,  development  of  personal  and  institutional  experience,  and  the  ability  to 


1  The  term  high-reliability  organizations  (HROs)  is  used  by  Weick  and  Sutcliffe  to  describe  organizations  that 
have  “no  choice  but  to  function  reliably.  If  reliability  is  compromised,  severe  harm  results”  (Weick  and  Sutcliffe, 
2007).  Specific  examples  of  HROs  include  nuclear  power  plants,  air  traffic  control  systems,  and  hostage  negotia¬ 
tion  teams. 
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leverage  historical  analogies — are  important  elements  of  an  effective  preparation  and 
response  plan.  However,  as  the  starting  point  of  our  own  research,  we  needed  to  under¬ 
stand  the  specifics  of  the  decisionmaking  process  that  professionals  use  when  con¬ 
fronting  surprise.  This  required  identifying  a  relevant  model  as  a  starting  point  from 
which  to  develop  testable  hypotheses  to  probe  through  our  discussions  with  expert 
practitioners. 

In  the  end,  we  settled  on  Gary  Klein’s  Sources  of  Power:  How  People  Make  Deci¬ 
sions  (1998).  Through  years  of  field  observations  watching  firefighters,  aircraft  carrier 
operations,  and  intensive  care  units,  Klein  has  refined  what  he  calls  the  recognition- 
primed  decision  model  (RPD).  The  RPD  provides  a  notional  framework  for  how  high- 
stakes  decisions  are  made  (Klein,  1998,  p.  7). 2  As  Klein  notes  in  his  exposition  of  the 
model,  it  fuses  two  processes:  how  decisionmakers  use  prior  experience  to  recognize 
a  situation  and  how  decisionmakers  decide  on  an  appropriate  course  of  action  (Klein, 
1998,  p.  24). 

Klein’s  model  outlines  three  potential  paths  that  decisionmakers  follow  when 
confronting  a  decision: 

•  The  first  is  for  executing  prescribed  responses  to  easily  identified  problems:  The 
decisionmaker  recognizes  the  change  as  something  that  has  been  seen  before  and 
proceeds  using  a  previously  prescribed  course  of  action. 

•  The  second  variant  is  for  when  the  problem  cannot  be  immediately  identified. 
In  this  case,  the  practitioner  will  attempt  to  diagnose  the  situation  by  matching 
observed  features  with  past  experience.  If  this  is  not  immediately  successful,  the 
practitioner  continues  to  observe  the  scenario  to  collect  more  data  until  he  or  she 
is  able  to  determine  causality. 

•  The  third  RPD  variant  applies  when  the  practitioner  can  identify  the  problem  but 
does  not  have  a  prescribed  solution  at  hand.  In  this  case,  the  decisionmaker  often 
uses  mental  simulation  to  evaluate  each  potential  response  to  identify  the  best 
option.  Once  the  optimal  solution  is  identified,  the  decisionmaker  can  implement 
that  as  the  proper  course  of  action. 

Klein  provides  a  helpful  analogy  for  understanding  these  paths,  likening  deci¬ 
sionmaking  to  an  “if-then”  process.  The  first  variant  is  when  the  decisionmaker  knows 
both  the  “if”  and  the  “then.”  The  second  is  when  the  decisionmaker  only  knows  the 
“then,”  and  the  third  is  when  he  or  she  only  knows  the  “if”  (Klein,  1998). 

Klein’s  model  therefore  provides  an  initial  framework  that  describes  how  pro¬ 
fessionals  make  decisions  in  uncertain  environments.  The  model  suggests  three  key 
aspects  regarding  how  people  respond  to  surprise: 


2  Klein  implicitly  defines  high-stakes  decisions  as  decisions  in  which  lives  or  significant  resources  are  at  stake 
(Klein  1998,  p.  4). 
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•  Whether  conscious  or  subconscious,  decisionmakers  rely  on  a  notional  decision 
loop  to  evaluate  and  make  decisions. 

•  Decisionmakers  will  use  different  mental  mechanisms  to  respond,  depending  on 
whether  the  surprise  is  immediately  recognized. 

•  When  confronted  with  a  situation  for  which  they  lack  a  prescribed  solution,  deci¬ 
sionmakers  use  mental  simulation  to  test  out  potential  responses  before  deciding 
on  a  course  of  action. 

We  were  also  interested  in  probing  deeper  into  the  nature  of  the  surprise,  so  we 
drew  inspiration  from  nine  factors  that  Klein  uses  to  define  a  naturalistic  decision¬ 
making  setting  (2008).  These  factors  are  experienced  decisionmakers,  high-stakes  sce¬ 
narios,  dynamic  conditions,  inadequate  information,  time  pressure,  ill-defined  goals, 
poorly  defined  procedures,  team  coordination,  and  cue  learning.  Because  our  research 
objective  required  distinguishing  between  various  surprise  operating  environments, 
we  focused  on  three  key  factors  of  naturalistic  decisionmaking  settings:  time  pressure, 
inadequate  information,  and  dynamic  conditions.  Those  three  factors  appeared  (to  us) 
most  likely  to  influence  the  approach  that  experienced  professionals  use  when  respond¬ 
ing  to  unexpected  events. 
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We  Developed  a  Framework  to  Group 
and  Classify  Professions 
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We  refined  Klein’s  factors  of  time  pressure,  inadequate  information,  and  dynamic  con¬ 
ditions  into  a  two-dimensional  classifying  framework,  or  “surprise  space.”  The  frame¬ 
work  that  we  developed  is  shown  above.  The  two  axes  are  categorized  by  typical  response 
time  and  level  of  environmental  chaos  (which  combines  the  concept  of  a  dynamic  envi¬ 
ronment  with  one  of  inadequate  information). 

We  used  typical  response  time  to  characterize  the  horizontal  axis,  and  we  have  four 
discrete  points  (seconds,  minutes,  hours,  and  days  and  weeks)  that  characterize  an 
occupation’s  most  common  operating  mode.3  An  example  of  someone  who  operates 
with  a  typical  response  time  of  a  few  seconds  is  a  combat  airplane  test  pilot;  an  example 
of  someone  who  operates  with  a  response  time  of  days  and  weeks  is  a  CEO. 

We  used  level  of  chaos  in  the  environment  to  characterize  the  vertical  axis,  and  we 
chose  three  levels:  low,  medium,  and  high.  We  define  chaos  as  a  subjective  measure  of 
the  frequency,  diversity,  and  predictable  orderliness  of  events,  and  this  meaning  seemed 
to  best  fit  the  quality  that  we  were  trying  to  assess.  This  definition  followed  from  the 
recognition  that  some  professions  work  in  more  controlled  environments  than  others, 
and  we  hypothesized  that  this  would  be  an  important  factor  when  considering  how 


3  We  characterize  response  time  as  the  span  between  the  surprise  being  detected  and  the  point  at  which  a  deci¬ 
sion  must  be  made  to  mitigate  or  prevent  the  event  from  becoming  a  more-challenging  scenario. 
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professions  approach  and  respond  to  surprise.  Our  definition  incorporates  that  inabil¬ 
ity  to  predict  or  control  events  but  also  includes  a  measure  of  how  frequently  those 
events  occur  and  how  disorderly  the  resulting  environment  is. 

Those  occupations  working  in  low-chaos  environments  are  jobs  that  take  place 
in  very  contrived  environments.  An  NFL  coach  is  a  good  example  of  someone  who 
works  in  a  low-chaos  environment  because  he  or  she  works  in  an  athletic  arena,  where 
the  grounds  are  carefully  manicured,  the  climate  is  often  controlled,  and  the  fans  are 
separated  from  the  players  by  physical  barriers.  A  heart  surgeon  is  a  good  example  of 
someone  who  works  in  a  moderately  chaotic  environment.  The  surgeon  works  in  a 
semi-contrived  environment:  Operating  rooms  are  sterilized,  full  of  instrumentation, 
well  lit,  and  climate  controlled.  However,  the  surgeon  faces  more  complexity  than  the 
NFL  player  because  a  greater  number  of  factors  can  influence  the  situation.  An  exam¬ 
ple  of  someone  who  works  in  a  highly  chaotic  environment  is  any  profession  that  works 
mainly  “in  the  field,”  where  the  environment  cannot  easily  be  controlled.  For  example, 
Special  Weapons  and  Tactics  (SWAT)  teams  must  deal  with  civilians,  criminal  perpe¬ 
trators,  pets,  equipment,  weather,  and  other  law  enforcement  colleagues.  All  of  these 
factors  contribute  to  what  we  are  calling  “highly  chaotic”  environments. 

Our  goal  in  developing  this  framework  was  to  ensure  that  we  considered  a  broad 
spectrum  of  interviewees  (professions)  based  on  the  two  characteristics  that  we  describe 
above.  We  recognized  from  the  outset  that  we  did  not  have  the  resources  to  conduct  a 
comprehensive  study  on  this  subject.  Instead,  we  developed  the  framework  because  we 
were  looking  for  a  simple  method  that  we  could  use  to  quickly  draw  lessons  about  how 
different  classes  of  practitioners  use  different  methods. 
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We  Populated  Our  Framework  with  Jobs  of 

Different  Response  Times  and  Levels  of  Chaos 
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After  developing  the  framework,  we  populated  the  axes  with  a  list  of  example  profes¬ 
sions.  Our  goal  in  populating  the  axes  was  to  identify  at  least  one  occupation  for  each 
intersection.4  (We  will  defer  discussing  our  selection  method  for  a  moment  and  will 
first  describe  the  research  hypotheses  that  we  wanted  to  test.) 

The  framework  shown  in  this  slide  was  particularly  helpful  because  we  used  it  to 
motivate  the  hypotheses  for  our  research.  For  example,  our  first  research  question  came 
from  looking  at  all  of  the  professions  shown  in  this  slide:  Are  there  preparatory  and 
surprise  techniques  that  are  common  to  all  (or  most)  occupations? 


4  We  chose  not  to  populate  the  lower  left  corner  of  the  framework.  The  goal  of  this  research  was  to  identify  ways 
to  become  more  agile  and  flexible,  and  we  assumed  that  occupations  operating  in  a  low  level  of  environmental 
chaos  with  hours,  days,  and  weeks  of  response  time  would  not  yield  satisfying  conclusions  toward  our  research 
objective. 
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We  Observed  Two  Groups  Side  to  Side: 
Tactical  vs.  Strategically  Focused  Occupations 
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In  developing  our  second  hypothesis,  we  observed  that  splitting  the  visual  framework 
down  the  middle  of  the  chart  creates  two  groups.  Those  occupations  on  the  right  side 
of  the  framework  (in  yellow  and  orange)  are  all  very  tactical  in  nature,  so  we  started 
referring  to  this  group  as  the  tactical  professions,  using  the  military’s  connotation  of  the 
word  tactical .* 5  One  observation  about  most  of  the  tactical  professions  is  that  they  are 
skilled  in  touch  labor:  They  work  with  their  hands.  This  is  likely  due  to  the  fact  that  all 
of  these  professionals  need  to  get  things  done  quickly,  and  personally  interacting  with 
their  environment  is  the  best  way  to  be  most  responsive. 

By  contrast,  those  occupations  on  the  left  side  of  the  framework  (in  blue  and 
green)  are  all  more  strategic  in  nature,  with  practitioners  often  having  hours,  days, 
or  weeks  to  respond  to  unexpected  events.  We  started  referring  to  this  group  as  the 
strategic  professions.  Workers  in  these  occupations  are  also  referred  to  as  “knowledge 
workers,”  and  they  are  valued  for  the  knowledge  capital  that  they  contribute  to  their 
respective  organizations.6 


6  In  military  usage,  a  tactical  operation  is  one  that  requires  or  involves  near-term  (short  time  frame)  support 

actions. 

6  By  associating  the  term  knowledge  workers  with  the  more  strategically  oriented  professions,  we  inadvertently 
suggest  that  the  tactical  professions  are  not  valued  for  their  own  knowledge  capital.  This  is  certainly  not  our 
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Finally,  in  using  the  terms  strategic  and  tactical  for  occupations,  we  recognize  that 
those  individuals  working  in  more-tactical  occupations  certainly  do  engage  in  long¬ 
term  (strategic)  planning  as  well,  and  vice  versa  for  the  strategic  professions.  We  use  the 
terms  strategic  and  tactical  to  indicate  each  profession’s  most  common  operating  mode 
when  responding  to  unexpected  events. 

This  observation  of  tactical  and  strategic  professions  motivated  our  second 
hypothesis:  Do  the  tactical  professions  respond  to  unexpected  events  using  different 
methods  than  those  of  the  strategic  professions? 


intent.  We  chose  to  use  the  term  knowledge  worker  because  it  is  a  phrase  that  is  already  part  of  the  vernacular,  and 
the  colloquial  meaning  is  in  convenient  alignment  with  our  strategic  professions. 
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We  Also  Observed  Two  Groups  Top  to  Bottom: 

Complex  vs.  More  Contrived  Environments 
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Our  third  hypothesis  is  based  on  an  observation  that  the  occupations  in  the  most  cha¬ 
otic  environments  differ  from  those  in  the  other  two  rows.  We  observed  that  all  of  the 
occupations  operating  in  the  most  chaotic  environments  all  face  surprise  that  is  gener¬ 
ated  by  other  humans  most  of  the  time.  By  contrast,  all  of  the  professions  working  in 
moderately  chaotic  workspaces  face  surprises  that  are  (generally)  not  caused  by  human 
actions,  and  this  results  in  a  less  chaotic  environment.7 

These  observations  motivated  our  third  hypothesis:  Does  the  level  of  environmen¬ 
tal  chaos  affect  the  way  practitioners  prepare  for  unexpected  events? 


7  The  NFL  coach  and  improv  actor  do  face  surprises  that  are  generated  by  other  humans,  so  why  do  we  list  them 
as  working  in  a  low-chaos  environment?  The  reason  is  that  there  are  two  contradictory  processes  at  work:  These 
professions  face  surprise  that  is  usually  generated  by  other  humans,  but  they  also  work  in  very  structured  environ¬ 
ments,  which  constrains  everyone’s  decisions  and  actions.  In  addition,  there  are  no  third  parties  in  NFL  football 
or  improvisational  theater  (save  for  the  spectators).  Thus,  while  unconstrained  humans  seem  able  to  generate  the 
most-chaotic  surprise  environment  of  any  we  studied,  the  constraints  of  a  structured  environment  and  paucity  of 
third-party  actors  cancel  this  factor  out. 
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We  Developed  Hypotheses  and  Sought  Out 
Practitioners  to  Test  Our  Initial  Assumptions 

•  Our  key  hypotheses  were  based  on  observations 
made  using  our  framework: 

1.  Are  there  response  techniques  common  to  all 
practitioners? 

2.  Does  the  response  depend  on  timescale? 

3.  Does  the  response  depend  on  the  level  of 
chaos  in  the  environment? 

•  We  conducted  discussions  with  experts  to  test  our 
hypotheses: 

-  Did  we  get  the  correct  response  time/chaos? 

-  What  are  some  examples  of  surprise? 

RAND 


Once  we  developed  our  key  research  questions,  we  took  a  methodical  approach  to 
answering  them.  Our  aim  was  to  span  the  spectrum  of  surprise  environments  by  select¬ 
ing  an  occupation  for  each  X-Y  intersection.  We  made  these  selections  based  on  sev¬ 
eral  criteria.  The  first  factor  was  entirely  due  to  practicality:  We  had  limited  time  and 
resources  for  this  project,  so  we  wanted  to  select  professions  for  which  we  were  confi¬ 
dent  that  we  could  readily  recruit  and  speak  with  an  experienced  practitioner.  We  also 
viewed  the  variety  of  professions  as  an  asset  toward  meeting  our  research  objective.  By 
interviewing  a  wide  variety  of  practitioners  (instead  of  just  health  care  providers,  for 
example),  we  hoped  to  gain  a  wider  variety  of  backgrounds  and  perspectives.  Indeed, 
as  we  will  describe,  this  ended  up  being  an  asset  when  we  heard  similar  themes  emerge 
from  seemingly  different  perspectives:  It  helped  reinforce  the  notion  that  the  response 
strategies  we  were  hearing  were  robust  and  could  be  used  across  multiple  professions. 

In  addition  to  these  primary  criteria,  our  participating  practitioners  had  to  have 
at  least  ten  years  of  post-training  professional  experience,  have  achieved  a  high  supervi¬ 
sory  or  equivalent  level  of  responsibility,  and  have  otherwise  distinguished  themselves 
(through  commendations,  awards,  or  peer  recognition)  as  superior  performers  in  their 
fields.  We  imposed  these  additional  criteria  because  we  assumed  that  speaking  with 
experienced  and  successful  individuals  would  be  the  most  efficient  way  to  learn  effec¬ 
tive  and  best  practices. 
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We  were  fortunate  in  that  everyone  with  whom  we  attempted  to  speak  agreed  to 
participate  in  the  project.  The  final  list  included  13  representative  occupations:  CEO, 
retired  U.S.  ambassador,  test  pilot,  foreign  service  officer,  cardio thoracic  surgeon, 
recently  retired  Navy  SEAL,  recently  retired  NFL  coach,  professional  improv  actor, 
public  works/civil  engineer,  space  mission  planner/operator,  recently  retired  Air  Force 
lieutenant  general,  SWAT  team  commander,  and  emergency  room  (ER)  physician. 

We  have  refrained  from  disclosing  the  names  of  those  with  whom  we  spoke 
because  part  of  our  research  approach  was  to  promise  our  participants  anonymity. 
Conversations  lasted  from  45  to  90  minutes,  with  approximately  half  conducted  by 
phone  and  the  other  half  in  person.  Two  researchers  participated  in  each  discussion, 
with  one  taking  the  lead  role,  and  the  other  primarily  taking  notes. 

We  had  three  goals  for  these  discussions.  First,  we  wanted  to  confirm  that  we  had 
correctly  located  each  profession  within  our  axes.  To  do  this,  we  asked  each  represen¬ 
tative  how  quickly  he  or  she  had  to  react  when  responding  to  surprise.  We  also  asked 
them  questions  about  their  operating  environments  to  make  sure  that  our  estimate 
about  their  level  of  chaos  was  accurate.  The  positions  shown  on  the  previous  slides 
reflect  the  final,  representative  locations  of  all  the  exemplar  occupations. 

The  second  goal  of  the  discussions  was  to  learn  more  about  the  techniques  and 
tools  each  person  used  when  responding  to  surprise.  To  do  this,  we  asked  the  par¬ 
ticipants  questions  about  how  they  typically  responded  to  surprise  and  what  (if  any) 
professional  or  organizational  protocols  they  relied  on  when  responding.  Our  ques¬ 
tions  were  based  on  our  hypotheses  and  research  objectives,  but  the  conversations  were 
allowed  to  progress  organically,  and  they  did  not  follow  a  highly  structured  protocol. 

The  final  goal  of  these  discussions  was  to  obtain  anecdotes  that  could  be  used  to 
support  and  illustrate  our  findings.  Participants  offered  a  surprisingly  rich  and  varied 
set  of  such  stories,  some  of  which  we  share  in  the  following  pages. 

We  conducted  semistructured  interviews  with  practitioners  working  in  each  of 
these  professions  to  achieve  the  three  goals  mentioned  above.  Before  beginning  the 
interviews,  we  developed  a  basic  protocol  that  outlined  the  key  topics  that  we  hoped 
to  discuss  in  every  conversation.  Specifically,  we  focused  on  the  following  (exemplar) 
questions: 

•  What  is  the  level  of  chaos  and  usual  available  response  time  in  the  typical  operat¬ 
ing  environment? 

•  What  are  some  other  parameters  and  constraints  of  the  typical  operating  environ¬ 
ment? 

•  What  are  the  criteria  for  operational  success  or  failure? 

•  What  role  does  surprise  play  in  the  operating  environment? 

•  What  does  the  response  process  look  like  when  surprises  occur? 

•  What  resources,  tools,  and  strategies  are  available  for  dealing  with  surprise,  and 
how  are  they  typically  applied? 


46  Increasing  Flexibility  and  Agility  at  the  National  Reconnaissance  Office 


•  How  much  do  flexibility  and  finesse  apply  when  dealing  with  surprise? 

•  To  what  extent  are  surprises  viewed  as  opportunities  rather  than  obstacles? 

•  What  are  the  key  elements  needed  to  successfully  deal  with  surprise?  What  are 
some  examples  of  when  those  approaches  failed? 

•  What  are  some  characteristics  of  a  professional  in  the  field  that  allow  the  person 
to  best  handle  surprise? 

•  Finally,  we  always  gave  participants  the  opportunity  to  express  additional  thoughts 
on  topics  that  had  not  been  covered  earlier. 

Data  from  each  practitioner  conversation  was  analyzed  using  qualitative  meth¬ 
ods.  Both  investigators  discussed  the  content  and  debriefed  each  other  within  24  hours 
following  each  conversation.  We  compared  information  provided  by  the  professional 
against  the  relevant  components  of  our  initial  conceptual  model,  looking  for  confir¬ 
mation,  disconfirmation,  and  novel  insights  or  features.  We  used  the  descriptions  that 
participants  described  about  their  methods  and  processes  to  test  our  hypotheses. 

We  also  compared  their  information  against  that  provided  by  other  professionals, 
looking  for  subjective  similarities  and  differences  in  their  approaches  to  surprise.  The 
goal  of  this  particular  exercise  was  to  identify  key  themes  across  the  different  practitio¬ 
ners.  Where  differences  in  approach  could  be  identified,  we  also  attempted  to  charac¬ 
terize  any  associated  differences  in  the  surprise  environment  between  those  professions. 
These  iterative  updates  to  our  initial  hypotheses,  including  any  tentative  conclusions 
regarding  the  reasons  for  differences  and  similarities  in  approaches  between  profes¬ 
sions,  were  then  subsequently  tested  when  discussing  related  topics  with  other  profes¬ 
sionals.  Following  the  completion  of  our  discussions  with  all  recruited  professionals, 
the  results  of  this  analysis  were  then  organized  and  grouped  by  initial  hypothesis. 

It  was  not  our  intent  to  evaluate  our  interview  data  using  quantitative  or  statisti¬ 
cal  means.  For  example,  we  did  not  set  out  to  make  statements  like,  “75  percent  of  all 
CEOs  rely  on  the  same  method  to  respond  to  an  unexpected  event.”  This  is  because 
such  an  approach  was  not  in  alignment  with  our  project’s  goal  of  simply  identifying 
general  trends  across  the  professions.  A  second  reason  is  that  we  interviewed  15  profes¬ 
sionals  in  total,  which  is  not  a  statistically  large  sample. 

How  did  we  know  that  interviewing  15  professionals  was  sufficient  in  meeting 
our  goal  for  identifying  general  trends?  For  guidance  on  this,  we  followed  a  sugges¬ 
tion  by  Robert  S.  Weiss  in  Learning  from  Strangers:  The  Art  and  Method  of  Qualitative 
Interview  Studies  (1994):  “[Y]ou  stop  when  you  encounter  diminishing  returns,  when 
the  information  you  obtain  is  redundant  or  peripheral.”  About  two-thirds  of  the  way 
through  the  interview  process,  we  began  to  recognize  familiar  narratives,  a  good  indi¬ 
cation  that  our  small  data  set  was  nevertheless  providing  empirically  reliable  data.  This 
suggested  that  we  had  reached  what  survey  managers  call  the  “saturation  point”  and 
was  a  good  indication  that  the  number  of  interviews  was  sufficient  for  meeting  our 
research  objective. 
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We  Found  That  Some  Response  Techniques 
Were  Common  to  All  Practitioners 

•  We  observed  that  all  successful  surprise  practitioners: 

-  Rely  heavily  on  past  experience,  including 
experience  gained  via  repetitive  training. 

-  Seek  to  reduce  the  level  of  chaos  within  the 
operating  environment 

-  Match  the  response  level  to  the  actual  need 

-  Consistently  mention  that  teamwork  is  an  essential 
component 


RAND 


We  found  a  number  of  coping  strategies  that  were  common  across  all  of  the  professions. 

First,  we  observed  that  all  practitioners  rely  on  experience.  Experience  is  one  of 
the  best  insurance  policies  against  the  negative  effects  of  surprise  because  it  provides 
the  context  necessary  to  recognize  and  respond  to  unexpected  events.  Experienced 
practitioners  are  able  to  quickly  identify  surprises  and  generate  potential  reactions  and 
solutions  based  on  what  they  have  done  in  the  past. 

We  also  observed  that  all  professions  try  to  reduce  the  level  of  chaos  in  the  oper¬ 
ating  environment,  since  reducing  chaos  also  reduces  the  complexity  and  size  of  the 
solution  space.  Reducing  chaos  can  be  done  in  different  ways,  either  by  controlling  the 
environment  or  spending  a  few  extra  moments  (or  days,  depending  on  the  profession) 
gathering  more  data  on  the  situation. 

For  example,  test  pilots  only  execute  one  maneuver  at  a  time.  In  addition,  the 
test  plane  is  outfitted  with  a  suite  of  environmental  sensors  that  are  constantly  being 
monitored  by  a  room  of  test  engineers.  When  the  engineers  detect  a  problem,  they  are 
able  to  provide  the  pilot  with  data  and  context  that  help  him  to  focus  his  response  pro¬ 
cedures  toward  the  root  cause  of  the  problem. 

Similarly,  the  cardiothoracic  surgeon  with  whom  we  spoke  prefers  to  work  with 
the  same  team  of  nurses  and  doctors  for  all  of  his  procedures.  In  addition,  he  organizes 
his  surgical  tools  using  the  same  layout  for  every  surgery,  and  he  tends  to  repeatedly  use 
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pieces  of  monitoring  equipment  with  which  he  is  most  familiar.  Both  of  these  examples 
demonstrate  ways  that  practitioners  reduce  the  size  of  the  surprise  space  to  minimize 
the  number  of  things  that  can  go  wrong. 

We  also  learned  that  it  is  best  to  react  to  surprises  with  a  measured  response  to 
preserve  further  options  as  the  surprise  unfolds.  As  examples  of  this,  both  the  CEO 
and  the  Navy  SEAL  used  essentially  the  same  language  to  tell  us  that  it  is  important 
not  to  overreact  when  confronted  by  a  surprise.  Along  with  nearly  all  of  our  partici¬ 
pants,  they  emphasized  the  importance  of  matching  the  remedy  to  the  actual  need. 

Finally,  we  observed  that  teamwork  plays  an  essential  role  when  responding  to 
unexpected  events,  even  for  those  professions  that  are  usually  perceived  as  relying  on 
individual  actors,  like  heart  surgeons  or  test  pilots.  All  of  the  professions  emphasized 
that  they  all  relied  heavily  on  teamwork  to  successfully  deal  with  surprise  events. 
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We  Observed  That  the  Level  of  Environmental 
Chaos  Determines  How  Practitioners  Prepare 

*  In  the  most  contrived  environments,  practitioners 
develop  and  rely  on  specific  “what  if”  plans 

*  In  moderately  complex  environments,  practitioners 

-  Prepare  “what  if”  plans  for  likely  and  existential 
threats 

-  Rely  on  protocols  and  training  for  all  else 

*  In  complex  environments,  practitioners 

-  Develop  robust  and  generalized  response 
frameworks  (and  exercise  them) 


RAND 


We  found  that  the  level  of  environmental  chaos  strongly  affects  how  practitioners  pre¬ 
pare  for  and  respond  to  surprise.  Specifically,  we  found  that  practitioners  who  work  in 
the  most  structured  (low-chaos)  environments,  like  athletic  fields  or  theatrical  stages 
in  which  most  environmental  factors  are  controlled,  face  only  a  finite  range  of  possible 
outcomes.  Because  of  this,  these  practitioners  are  able  to  plan  reactions  for  nearly  any 
event,  and  practitioners  in  the  NFL,  for  example,  regularly  do  so.  For  many  working 
in  low-chaos  environments,  it  makes  sense  to  develop  comprehensive  “what  if”  plans 
because  the  size  of  the  surprise  space  is  small  (compared  to  other  professions)  and  finite. 

Practitioners  who  work  in  moderately  chaotic  environments  like  operating  rooms 
or  test  plane  cockpits  rely  on  preplanned  protocols  for  the  most  likely  events,  along 
with  anything  that  represents  an  existential  threat.  As  an  example,  the  ER  physician 
with  whom  we  spoke  indicated  that  he  had  protocols  to  deal  with  some  of  the  most 
common  injuries  that  arrive  in  the  emergency  department.  However,  he  also  recog¬ 
nized  that  there  are  too  many  unforeseen  events  in  moderately  chaotic  environments 
to  plan  against  every  possibility,  so  he  relied  on  basic  response  frameworks  when  the 
surprise  event  was  not  covered  by  a  specific  protocol. 

In  addition  to  preparing  for  the  most  common  threats,  we  found  that  practitio¬ 
ners  in  moderately  complex  environments  also  make  “what  if”  plans  for  existential 
threats.  For  example,  the  Mars  rovers  take  most  of  their  commands  directly  from 
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mission  managers  on  Earth  each  day.  However,  a  small  number  of  commands  are 
embedded  (preprogrammed)  on  the  spacecraft  to  guard  against  existential  threats.  An 
example  of  a  mission-ending  threat  would  be  if  the  rover’s  communications  dish  were 
ever  positioned  below  the  horizon  line.  If  this  occurred,  the  rover  would  not  be  able 
to  receive  commands  from  earth,  and  this  would  effectively  represent  the  end  of  the 
rover’s  life.  To  address  this  possibility,  the  rovers  are  preprogrammed  with  instructions 
to  automatically  reposition  the  antenna  back  toward  Earth. 

The  most  challenging  circumstances  are  faced  by  those  practitioners  working  in 
highly  chaotic  environments,  such  as  in  a  foreign  embassy  or  behind  enemy  lines. 
Their  environments  are  so  complex  and  unpredictable  that  it  does  not  make  sense  to 
do  much  planning  against  specific  surprise  events.  (This  is  because  workers  in  the  held 
must  deal  with  third  parties,  unpredictable  weather,  and  unfamiliar  terrain.)  Instead, 
practitioners  working  in  highly  chaotic  environments  develop  and  exercise  a  general- 
purpose  framework  that  can  be  deployed  whenever  a  major  surprise  is  encountered. 

As  an  example,  the  former  ambassador  with  whom  we  spoke  referred  to  his 
framework  as  a  “task  force”  and  noted  that  the  task  force  is  the  standard  tool  that  he 
used  whenever  he  encountered  a  surprise.  Therefore,  he  and  his  staff  had  worked  out 
the  details  of  assembling  a  task  force  ahead  of  time.  While  they  did  not  know  what 
the  specific  surprise  was  going  to  be,  they  knew  it  would  require  office  space,  lines  of 
communication,  and  the  support  of  some  key  people  inside  and  outside  the  embassy. 
They  then  prepared  and  practiced  a  process  for  quickly  deploying  this  infrastructure 
when  needed. 
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We  Observed  That  Strategists  Use  Different 
Response  Mechanisms  Than  Tacticians 

•  When  a  surprise  happens  to  a  tactician,  they  are  often 
trying  to  control  fear  and  anxiety.  Their  response  tends 
to  involve  the  following  steps: 

-  Control  panic 

-  Buy  time 

-  Revert  to  fundamentals  learned  in  training,  allowing 
effective  response  using  minimal  analysis. 

•  By  contrast,  strategists  often  have  to  control  ego,  anger, 
and  overreaction.  To  do  this,  effective  strategists  will 

-  Control  emotions  to  maintain  objectivity 

-  Take  initial  enabling  actions 

-  Assemble  the  staff 

-  Socialize  the  longer-term  plan 
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We  found  that  practitioners  within  our  tactical  professions  rely  on  different  tools  than 
those  within  our  strategic  professions. 8 

Based  on  the  data  that  we  collected  during  our  interviews,  we  found  that  when 
tacticians  are  surprised,  they  often  have  to  deal  with  emotions  of  fear  and  anxiety.  As 
a  result,  we  found  that  their  response  loop  is  designed  to  combat  those  emotions.  We 
also  note  that  tacticians  typically  rely  on  protocols  to  overcome  these  emotions  in  order 
to  respond  effectively  using  minimal  analysis.  As  an  example,  the  test  pilot  noted  that 
he  was  trained  to  “wind  the  watch”  in  the  moments  after  a  surprise,  which  refers  to  the 
menial  task  of  winding  the  stem  on  his  mechanical  wristwatch.  The  goal  of  this  task 
is  to  focus  on  a  basic  manual  chore,  with  the  hope  that  it  would  provide  calmness  and 
clarity  in  the  immediate  moments  after  a  surprise  occurs.  Likewise,  the  former  Navy 
SEAL  noted  that  when  SEAL  teams  are  fired  upon,  they  quickly  take  steps  designed 
to  control  panic  and  buy  time:  They  fall  into  rehearsed  positions  on  the  ground  that 
help  them  establish  a  strong  defensive  position.  In  taking  these  positions,  the  team  is 


8  Again,  we  use  the  words  tactical  and  strategic  with  specialized  meanings  in  this  document:  The  tactical  pro¬ 
fessions  are  those  that  have  to  typically  respond  within  minutes  or  seconds.  Members  of  the  strategic  professions 
typically  have  days,  weeks,  or  months  to  respond.  As  we  stated  earlier,  we  recognize  that  those  individuals  work¬ 
ing  in  more  tactical  occupations  certainly  also  engage  in  long-term  (strategic)  planning,  and  vice  versa  for  the 
strategic  professions. 
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able  to  gain  confidence  by  quickly  executing  a  familiar  task  that  they  have  performed 
hundreds  of  times  before. 

In  contrast  to  the  tacticians,  we  found  that  surprising  a  strategist  can  cause  anger 
and  impulse  desires  to  overreact.  To  combat  this,  we  found  that  many  successful  strat¬ 
egists  employ  the  same  four-step  process  to  counter  these  issues:  take  steps  to  control 
emotion  and  ego,  initiate  first-order  enabling  actions,  assemble  a  trusted  inner  circle  of 
advisors  and  direct  reports,  and  disseminate  a  coherent  longer-term  response  through¬ 
out  the  organization. 

The  CEO  gave  us  the  most  intuitive  example  of  a  way  to  control  emotion  and 
ego;  he  noted  that  instead  of  responding  immediately  when  surprised  by  an  email,  he 
places  the  message  in  his  drafts  folder  and  revisits  it  24  hours  later  to  confirm  that  the 
message  is  objective  and  constructive. 

The  retired  three-star  general  officer  gave  us  the  best  example  of  taking  initial 
enabling  actions  after  a  surprise  occurs.  The  general  with  whom  we  spoke  was  in  com¬ 
mand  of  some  key  logistics  resources  on  September  11,  2001.  As  the  day  unfolded,  no 
one  was  sure  what  was  going  on,  but  the  general  was  confident  that  he  would  be  called 
upon  to  provide  his  resources  to  the  national  command  authority.  Because  of  this,  he 
started  mobilizing  a  measured  number  of  resources  early  on  to  ensure  that  a  basic  level 
of  service  would  be  available.  This  proved  to  be  an  effective  decision  because  his  lead¬ 
ership  did  call  on  him  later  that  afternoon,  and  he  was  already  in  a  position  to  start 
responding  to  their  needs. 

As  we  have  already  noted,  most  strategists  are  valued  for  their  knowledge  capital, 
and  this  means  that  most  work  in  larger,  hierarchical  organizations.  Most  of  the  strate¬ 
gists  with  whom  we  spoke  were  senior  people  at  the  tops  of  their  respective  hierarchies, 
and  this  means  that  they  relied  heavily  on  direct  reports  and  trusted  staff  members 
whenever  a  surprise  occurred.  Therefore,  we  observed  that  the  next  step  strategists  took 
after  taking  steps  to  control  their  emotions  was  to  assemble  the  staff  and  develop  a  plan 
for  moving  forward.  In  doing  this,  all  of  the  strategists  noted  that  it  is  very  important 
to  decide  on  a  plan,  socialize  that  plan  throughout  the  organization,  and  present  a  uni¬ 
fied  front  when  executing  that  plan. 
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We  Also  Observed  Two  Key  Findings 
That  Were  Not  Part  of  Our  Initial  Hypotheses 

We  were  surprised  to  observe  that 

1.  The  most  chaotic  environments  all  feature 
surprises  that  are  motivated  by  other  humans 
rather  than  the  environment. 

2.  The  largest  surprises  come  from  third  parties 
rather  than  a  direct  stakeholder  or  adversary. 


RAND 


As  researchers,  we  were  surprised  by  two  findings  that  were  not  part  of  our  initial 
hypotheses.  First,  as  we  have  already  mentioned,  the  most  chaotic  environments  all  fea¬ 
ture  surprises  that  are  driven  by  other  humans  rather  than  the  operating  environment. 
Furthermore,  we  observed  that  the  biggest  surprises  tend  to  come  from  third  parties 
rather  than  a  direct  stakeholder  or  adversary. 

Our  initial  insight  into  this  idea  that  surprises  tend  to  come  from  third  parties 
came  from  the  Navy  SEAL,  who  noted  that  he  did  not  generally  encounter  major 
surprises  from  his  adversaries.  After  all,  he  had  spent  a  lot  of  time  thinking  about 
what  motivated  them,  along  with  determining  what  calculus  the  adversaries  were  likely 
using  to  achieve  their  goals.  Going  into  a  mission,  he  would  have  spent  time  thinking 
about  their  objectives  and  would  have  a  reasonably  good  chance  of  being  able  to  predict 
how  his  adversary  would  behave. 

So  what  surprises  the  Navy  SEAL  more  than  his  adversary?  Our  practitioner  told 
us  that  domesticated  animals  and  civilians  present  the  largest  surprises  to  his  team 
while  on  a  foot  patrol.  These  two  groups  represent  big  surprises  because  their  motiva¬ 
tions  and  actions  cannot  be  easily  predicted  by  the  SEALs.  Specifically,  civilians  are 
particularly  risky  for  his  team  to  encounter  because  the  SEALs  do  not  know  how  civil¬ 
ians  will  respond  to  them. 

These  two  observations  lead  to  our  recommendations. 
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Our  Research  on  Surprise  Yields 
Possible  Actions  for  NRO  Decisionmakers 

•  Finding:  experience,  organizational  memory,  level¬ 
headedness,  protocols,  and  teamwork  can  all  help 
mitigate  the  negative  effects  of  surprise. 

-  Possible  action:  Promote  knowledge  sharing 
through  a  common  AS&T  library,  web-based  sharing 
tools,  and  one-page  briefs  that  highlight  key  work. 

•  Finding:  the  biggest  surprises  to  strategists  often  come 
from  third-parties;  high-level  strategists  usually  are  not 
the  first  person  in  the  org  to  detect  the  surprise. 

-  Possible  action:  Focus  additional  resources  on 
entities  like  Africa,  the  Arctic,  and  NGOs. 

-  Possible  action:  Develop  a  Oth-order  model  that 
identifies  additional  third  parties. 

RAND 


To  conclude  this  section,  we  will  recap  our  findings  and  present  possible  actions  that 
NRO/AS&T  could  take  to  move  forward. 

First,  we  noted  that  experience,  controlling  emotions,  having  effective  protocols, 
and  relying  on  teamwork  can  all  help  mitigate  the  negative  effects  of  surprise.  There  are 
several  possible  actions  that  AS&T  could  take  based  on  these  findings: 

•  First,  AS&T  can  promote  knowledge-sharing  and  experience-sharing  through 
electronic  means  by  developing  a  central  repository  that  is  designed  to  facili¬ 
tate  information-sharing  throughout  the  organization.  This  could  be  achieved 
either  through  an  online  library  or  web-based  tools  that  help  employees  share 
their  experiences  with  one  another.  Recognizing  that  AS&T  analysts  are  often 
very  busy,  it  may  be  most  efficient  to  manage  knowledge-sharing  by  relying  more 
on  one-page  briefs  that  concisely  summarize  findings  rather  than  longer,  more- 
involved  reports. 

•  The  second  way  to  facilitate  experience-sharing  is  through  improved  human-to- 
human  interactions.  For  example,  multidisciplinary  core  teams  designed  to  tackle 
specific  problems  could  be  developed,  with  everyone  on  the  team  having  common 
security  credentials. 
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Our  second  key  finding  was  that  surprises  tend  to  come  from  third  parties,  and 
that  a  high-level  strategist  usually  is  not  the  first  person  in  the  organization  to  detect 
the  surprise.  (This  is  often  due  to  the  fact  that  senior  strategists  are  usually  located  near 
the  top  of  an  organizational  hierarchy.)  In  practice,  this  finding  suggests  two  possible 
actions  for  AS&T: 

•  First,  the  fact  that  the  biggest  surprises  are  most  likely  to  come  from  third  parties 
suggests  that  AS&T  should  focus  some  resources  toward  researching  third-party 
intelligence  problems.  Three  examples  of  these  third  parties  for  the  IC  might  be 
Africa,  the  Arctic,  and  nongovernmental  organizations  (NGOs),  which  are  play¬ 
ing  increasing  roles  in  world  politics  and  third-world  development.  In  addition, 
AS&T  might  also  benefit  by  considering  a  more  nuanced  version  of  this  lesson: 
What  third  parties  exist  in  threat  environments  that  are  already  established? 

•  Second,  AS&T  would  likely  benefit  from  investing  in  a  model  that  identifies 
additional  third  parties  that  may  pose  a  threat  in  the  future.  We  make  this  sug¬ 
gestion  with  the  knowledge  that  AS&T  invests  in  developing  “what  if”  plans 
against  existing  (known)  threats.  This  recommendation  suggests  investing  in 
models  that  can  identify  possible  threats  (or  threat  factors)  in  the  future.  After  all, 
simply  identifying  potential  threats  is  the  first  step  toward  mitigating  any  negative 
effects  that  may  be  caused  by  them. 


CHAPTER  FOUR 


Organizational  Mechanisms  That  Increase  Responsiveness 


This  Briefing  Summarizes  RAND’s  Research 
on  How  the  NRO  and  AS&T  Can  Become 

More  Agile 

Specific  research  questions: 


RAND 


For  the  last  section  of  this  report,  we  will  describe  some  pilot  research  that  looked  at 
how  other  organizations  run  their  research  and  development  (R&D)  processes. 
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Our  Objective  Was  to  Identify  Similar  Approaches 
Organizations  Use  to  Become  More  Responsive 


Company 

End  state 

Actions 

Became  more 
centralized 

•  Sold  off  unrelated  businesses 

Pfizer 

•  Closed  research  centers,  increased 

collaboration  with  universities 

IBM 

Shifted  from  products 

•  Sold  hardware  business  lines 

to  services 

•  Shifted  to  software/IT  consulting 

Became  more 
decentralized 

•  Created  self-contained  business  units 

Caterpillar 

•  Focused  on  increasing  flexibility  in  the 
workforce 

We  hypothesized  that  these  companies’  restructurings  would  provide 
lessons  on  becoming  more  competitive,  but  the  results  were  inconsistent. 

Why? 


RAND 


In  the  previous  two  sections,  we  examined  ways  that  the  NRO  could  allow  its  people 
and  hardware  to  become  more  flexible.  In  this  final  section,  we  performed  preliminary 
research  on  how  an  organization’s  processes  can  be  used  to  improve  responsiveness. 
Our  research  on  this  topic  was  motivated  by  questions  that  were  given  ro  us  by  AS&T 
personnel:  What  methods  have  other  organizations  used  to  become  more  responsive? 
Are  there  observable  similarities  that  could  benefit  the  NRO? 

To  do  this,  we  examined  three  companies  that  were  suggested  by  AS&T:  Pfizer, 
IBM,  and  Caterpillar.  These  companies  were  chosen  because  they  have  all  been  recently 
recognized  in  the  media  as  having  gone  through  transformations  in  order  to  better 
respond  to  pressures  in  the  marketplace. 

We  examined  these  companies  and  found  very  different  end  states  between  the 
three:  Pfizer  become  more  centralized,  IBM  started  selling  a  completely  different  prod¬ 
uct,  and  Caterpillar  become  more  decentralized  (Deviney  et  ah,  2012;  and  Zarroli, 
2011).  Why  did  these  companies  take  such  different  approaches? 
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We  Hypothesize  That  Different  Objectives  Will 
Require  Different  Approaches 

•  There  are  different  ways  to  become  more  responsive: 

-  Become  more  efficient  (do  more  with  less) 

-  Increase  agility  (quickly  adopt  new  technology) 

-  Pursue  the  state-of-the-art  (develop  a  brand  new 
capability) 

*  We  hypothesize  that  companies  take  different 
approaches  depending  on  their  objectives: 

-  Pfizer  reallocated  resources  to  pursue  high-risk, 
high-reward  goals. 

-  IBM  changed  their  business  line  to  avoid  an 
existential  threat. 

-  Caterpillar  reorganized  to  become  more  responsive 
to  customer  needs. 

RAND 


The  reason  for  this,  we  hypothesize,  is  that  there  are  different  ways  to  become  more 
responsive.  We  list  three  ways  on  this  slide,  but  there  are  certainly  more:  Some  com¬ 
panies  become  more  responsive  by  improving  efficiency — they  do  more  with  less.  We 
also  note  two  other  methods:  increasing  agility  (by  always  staying  current  with  new 
technologies,  for  example)  or  inventing  something  that  is  state  of  the  art.  The  reason  or 
reasons  driving  change  will  determine  the  methods  for  getting  there. 

For  example,  Pfizer  decided  to  concentrate  on  anticancer  and  Alzheimer’s  drugs 
(Thomas,  2012;  and  Wilson,  2011).  In  order  to  do  this,  it  sold  off  and  divested  all  of  its 
unrelated  properties  so  it  could  concentrate  on  this  high-risk,  high-reward  goal.  In  the 
process,  it  centralized  its  organization  and  processes  to  pursue  a  single  mission. 

By  contrast,  Caterpillar  was  interested  in  becoming  more  responsive  to  its  cus¬ 
tomers’  needs.  To  do  this,  it  decentralized  and  set  up  fully  contained  Caterpillar  offices 
around  the  country,  each  containing  everything  needed  to  run  the  business:  product 
experts,  sales  and  maintenance  teams,  and  finance  and  accounting  personnel.  In  doing 
this,  it  was  able  to  customize  its  service  to  the  local  market,  but  this  required  the  com¬ 
pany  to  take  a  different  approach  than  Pfizer. 

Finally,  IBM  took  yet  a  third  approach:  It  fundamentally  changed  its  product 
lines  by  choosing  to  focus  more  on  consulting  services  rather  than  primarily  delivering 
computing  hardware. 
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One  Future  Research  Objective  Would  Be  to  Match 

An  Organization’s  Goals  to  Specific  Processes 

•  Why  do  it? 

•  How  to  do  it? 

-  Cost  effective 

-  In  house  v. 

-  Reliability 

outsourced 

-  Agility 

-  Inline  R&D  v. 

-  State-of-the-art 

skunkworks 

-  Disruptive  products 

RAND 

-  Singular  v. 
multidisciplinary 
workforce 

-  Specific  v.  diverse 
research  portfolio 

We  observed  that  Pfizer,  Caterpillar,  and  IBM  each  took  a  different  approach  for 
becoming  more  responsive  in  their  responsive  marketplaces.  The  next  step  for  this 
research  would  be  to  develop  a  formal  framework  that  connects  the  methods  with  the 
specific  goals.  In  other  words:  Is  it  possible  to  develop  guidance  that  instructs  the  orga¬ 
nization  on  the  best  way  to  achieve  those  goals?  We  present  these  ideas  here  for  future 
researchers  who  may  be  interested  in  pursuing  these  topics. 

For  example,  we  recognize  that  there  are  several  decisions  that  organizations  have 
to  make  when  determining  how  to  become  more  responsive.  What  aspects  of  the  orga¬ 
nization  should  be  changed?  Should  R&D  be  performed  in  house,  or  should  it  be  out¬ 
sourced?  Should  the  company  rely  on  strategic  investments  for  new  intellectual  prop¬ 
erty,  or  should  it  develop  its  own  set  of  highly  skilled  innovators? 

In  addition,  how  should  the  research  department  be  organized?  Should  it  be 
designed  as  a  skunkworks,  in  which  a  few  of  the  most  capable  individuals  do  all  of  the 
work?  Or  should  the  company  democratize  the  process  by  giving  every  engineer  a  small 
percentage  of  free  time  to  work  on  pet  projects? 

Finally,  what  skills  should  the  overall  workforce  possess?  A  multidisciplinary 
workforce  is  more  likely  to  generate  a  wider  spread  of  ideas  (more  golden  nuggets  along 
with  more  rotten  eggs).  Alternatively,  a  singular  workforce  will,  on  average,  have  more 
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productive  ideas  than  the  multidisciplinary  force,  but  their  golden  nuggets  will  be  less 
likely  to  be  as  golden  as  those  from  the  multidisciplinary  team. 

We  did  not  have  the  resources  to  start  matching  the  reasons  for  innovating  with 
the  best  mechanisms  for  getting  there,  but  we  describe  this  issue  as  one  to  be  addressed 
in  future  work. 


CHAPTER  FIVE 

Conclusions 


This  Briefing  Summarizes  RAND’s  Research 
on  How  the  NRO  and  AS&T  Can  Become 

More  Agile 

Specific  research  questions: 


RAND 


We  conclude  this  report  by  making  some  observations  that  tie  all  three  topics  together, 
and  we  present  some  ideas  for  future  work. 
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Despite  the  Diversity  of  Research  Topics , 
There  Were  Common  Lessons  Throughout 

*  Observation:  Modularity  and  innovation  methods  are  not 
goals  by  themselves — they  are  tools. 

-  Takeaway:  Need  to  set  specific  priorities  and 
objectives  before  proceeding. 

*  Observation:  Success  in  modularity,  innovation,  and 
averting  surprise  requires  some  future  insight. 

-  Takeaway:  Invest  in  strategic  vision,  planning,  and 
exercises  that  probe  the  future. 

*  Observation:  Modularity,  surprise,  and  innovative 
processes  yield  ways  to  evolve  hardware,  people,  and 
corporate  structures,  respectively. 

-  Takeaway:  Becoming  more  agile  requires  an  effort  in 
all  three  areas  (hardware,  people,  and  corporate 
processes). 


RAND 


At  first  glance,  all  three  of  these  research  topics  appear  to  be  very  different,  but  we  did 
observe  several  common  lessons  throughout  this  project. 

The  first  observation  is  that  modularity  and  innovative  methods  are  not  goals  by 
themselves — they  are  tools  for  meeting  a  particular  goal.  No  one  sets  out  to  make  a 
modular  architecture  for  the  sake  of  making  a  modular  system.  A  modular  architecture 
is  motivated  by  a  set  of  user  needs,  which  usually  include  a  desire  to  develop  systems 
that  are  more  flexible  and  adaptable.  Similar  reasoning  holds  true  for  implementing 
innovative  methods:  They  are  always  used  to  achieve  a  larger  goal.  The  takeaway  from 
this  observation  is  that,  before  considering  modularity  or  new  innovative  practices,  the 
practitioner  needs  to  set  priorities  that  will  drive  how  to  proceed  using  modularity  and 
innovative  processes,  not  the  other  way  around. 

A  second  common  theme  is  that  being  successful  in  modularity,  innovation,  and 
averting  surprise  all  benefit  from  at  least  a  partial  ability  to  generate  future  insight. 
Therefore,  we  conclude  that  any  investments  in  developing  strategic  plans  or  visions, 
along  with  exercises  designed  to  probe  the  future,  can  benefit  all  three  topics. 

Finally,  we  noted  that  modularity,  surprise,  and  innovative  processes  yield  ways 
to  evolve  hardware,  people,  and  corporate  structures,  respectively.  Our  research  shows 
that  there  is  more  to  becoming  more  agile  and  responsive  than  just  developing  new 
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hardware.  Instead,  improved  responsiveness  comes  from  an  investment  in  all  three 
areas.  New  hardware  requires  an  equally  flexible  staff  to  design  and  operate  it. 
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Based  on  These  Common  Lessons , 

We  Propose  Three  Topics  for  Future  Research 

•  Finding:  A  wide  FOV  can  avert  surprise. 

-  Task:  Develop  a  wide  FOV  model  to  help  AS&T 
expand  their  “surprise  space.” 

•  Finding:  Modularity  can  limit  how  disruptive  a 
technology  could  be. 

-  Task:  How  do  engineered  systems  evolve  in 
disruptive  ways? 

•  Finding:  Organizations  innovate  for  different 
reasons. 

-  Task:  What  organizational  structures  are  best 
for  achieving  a  particular  goal? 

RAND 


We  conclude  this  report  by  proposing  three  topics  for  future  research. 

As  we  noted  earlier,  the  research  on  surprise  taught  us  that  adopting  a  wide  FOV 
can  help  mitigate  the  negative  effects  caused  by  surprise.  Therefore,  we  propose  devel¬ 
oping  a  first-order  wide  FOV  model  to  help  AS&T  expand  its  view  over  the  surprise 
space.  There  are  several  methods  that  could  be  used  to  address  this  task:  RAND’s 
Delphi  Method  and  RAND’s  existing  work  on  robust  decisionmaking  could  both  be 
used  to  generate  a  list  of  the  most  likely  third-party  actors  that  may  surprise  the  IC 
in  the  coming  years,  along  with  a  set  of  scenarios  that  represent  likely  outcomes  from 
these  surprises. 

Our  research  on  modularity  highlighted  the  fact  that  modular  systems  do  pose 
limits  on  how  disruptive  a  particular  technology  can  be.  Disruptive  technologies  create 
products  or  services  in  unexpected  ways,  and  modularity’s  frozen  interfaces  put  some 
restriction  on  how  much  a  product  is  able  to  change  in  the  future. 

Swiffer®  is  a  good  example  of  a  disruptive  technology.  Developed  by  Proctor  and 
Gamble,  Swiffer  uses  a  woven  dry  cloth  to  clean  floors  without  using  any  liquids. 
Before  Swiffer,  people  usually  cleaned  their  floors  with  a  mop,  a  bucket,  and  some  hot 
soapy  water.  When  Swiffer  was  being  tested  in  house,  Proctor  and  Gamble’s  experts  in 
cleaning  floors  noted  that  Swiffer  was  unlikely  to  succeed  because  it  did  not  perform 
as  well  as  conventional  wet  cleaning  methods.  This  is  certainly  true,  but  what  these 
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experts  failed  to  realize  was  that  many  consumers  had  never  wet-mopped  their  floors 
because  it  was  too  much  work  to  get  out  the  mop  and  the  bucket  of  water.  When 
Swiffer  was  introduced  to  the  market,  it  was  well  received  by  a  large  section  of  the 
population  that  had  never  mopped  before,  and  Swiffer  effectively  opened  a  new  market 
for  mopping  floors  using  dry  cleaning  methods. 

One  of  the  implied  goals  of  the  IC  is  to  create  and  field  more  disruptive  products. 
We  propose  performing  research  that  looks  to  the  biological  sciences  to  develop  a  set  of 
factors  that  highlight  the  conditions  needed  to  generate  disruptive  products.  For  exam¬ 
ple,  what  environmental  factors  have  led  to  some  of  nature’s  most  disruptive  patterns, 
like  wings  or  legs?  Or,  on  a  more  micro  level:  what  conditions  resulted  in  one  species 
gaining  more  colorful  plumage  than  a  neighboring  species?  After  examining  some  case 
studies,  we  propose  looking  to  see  what  the  engineering  community  can  learn  from 
these  patterns.  By  looking  at  some  analogs  in  nature,  we  hope  to  at  least  identify  the 
conditions  that  might  tend  to  generate  disruptive  products. 

Finally,  in  our  research  on  R&D  processes,  we  noted  that  different  organizations 
innovate  for  different  reasons.  We  propose  to  continue  this  research  by  investigating 
one  of  the  questions  that  we  posed  earlier  in  this  briefing.  Given  a  particular  reason 
for  innovating,  what  organizational  measures  are  most  likely  to  achieve  that  goal?  For 
example,  what  types  of  problems  is  a  skunkworks  best  at  solving?  When  would  it  be 
more  appropriate  to  conduct  “inline”  R&D  by  giving  every  engineer  10  percent  free 
time  to  pursue  pet  projects,  as  is  done  at  Google  and  3M? 
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